[HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Jarek Potiuk <ja...@potiuk.com>]

Hello here,

I have been thinking a lot recently and discussing with some people and I
am more and more convinced it's about the time we - as a community - should
start doing changes considering "Airflow 2" current and "Airflow 3" future.


*TL;DR: I think we should seriously start work on Airflow 3 and decide what
it means for our AIPs  - to treat some of them as more "tactical" - things
that should go into Airflow 2 and some "strategic" ones - being
foundational for Airflow 3 - with different goals and criteria.*

A lot of us already think that way and a lot of us have already talked
about it for quite some time, so you should treat my mail mostly as a
little trigger "let's start publicly discussing what it might mean for us
and our community and let's make it clear about the target of the
initiatives we do".

Some might be surprised it comes from me as I've been often saying "no
Airflow 3 without a good reason" or "possibly we will have no Airflow 3",
but I think (and a number of people I spoke to have similar opinion) we
have plenty of reasons to make some bold moves now.

Over the last 4 years since Airflow 2 was out, a lot has changed and we
have a number of different needs that current Airflow 2 cannot **really**
do well

- LLM/Gen-AI mainly as the important trigger
- Cloud Native is the "way to go"
- need to submit DAGs in other ways than dropping them to a shared DAG
folder.
- local testing and fast iteration on developing pipelines.
- ability to run tasks with workflow with "affinity" so that they can share
inputs/outputs in shared CPU/GPU memory
- ability to integrate seamlessly with other workflow engines - making
Airflow a "workflow of workflows
- probably way more
- all that while keeping a lot of the strengths of Airflow 2 - such as
continuing to have the option of using the many thousands of operators with
90+ providers.

All those above - we could implement better if we get rid of a number of
the implicit or explicit luggage we have in Airflow 2. I think the last two
proposals from Jens: AIP-68 and AIP-69 reflect very much that - both  would
have been much easier and straightforward if we got Airflow 3 re-designed
basically at a drawing board with boldly dropping some Airflow 3
assumptions.
And if we implemented core airflow 3 - taking the best part of what we have
now in Airflow 2, but generally dropping the luggage  in a new framework.

And it won't be possible without breaking some fundamental assumptions and
making Airflow 3 quite heavily incompatible with Airflow 2

From "my" camp - dropping the need of having the 700+ dependencies for
Airflow + all providers in a single Python interpreter, dropinnig
dependency on Flask/Plugins/FAB would be a huge win on its own. Not
mentioning being able to split provider's development and contribution from
airflow core (while keeping the development of providers as well and
contributions) - this has been highly requested.

And I think we have a lot of people in our community who would be able (and
would love) to do it - I think a number of us (including myself) are a bit
burned out and tired of just maintaining things in Airflow in a
backwards-compatible way and would jump on the opportunity to
rebuilding Airflow.

But - we of course cannot forget about Airflow 2 users. We do not want to
"stop the world" for them. We want to keep fixing things and adding
incremental changes - and those things do not necessarily super
"future-proof". They should help  to "keep the lights on" for a while -
which means that in a number of cases it could be "band-aid". AIP-44
(internal-API), AIP-67 (multi-team) are more of those.

So - what I think we might want to do as a community:

* start working on Airflow 3 foundations (and decide what it means for our
users and developer community). Decide what to keep, what to drop, what to
redesign, assumptions to recreate.

* explicitly split the initiatives/AIPs we have to target Airflow 2 and
Airflow 3 and treat them a bit differently in terms of future-proofness

I would love to hear your thoughts on that (bracing for the storm of those).

J.

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Elad Kalif <el...@apache.org>]

Hi everyone,
great discussion.

I talked with several community members and I think that we should have a
non technical discussion thread before we proceed.
To give an example:
We have several actors: Cluster Admin, Dag Author, UI user. In Airflow 2
perspective all these actors are the same user.
For example: If a user wants advanced scheduling, we will tell him that
Airflow has a solution for this - Timetables. However to deploy timetable
the Dag Author must interact with the Cluster Admin to deploy the plugin of
the timetable. These two actors can be two different persons with two
different agendas/road maps/priorities etc..
This is an example of something that we consider as solved - but is it
really solved?

I think before we dive deep into technical stuff we should first have a
vision of who the Airflow user(s), and what their needs/requirements are.
It should give us better direction, clarity and confidence that we are
developing the right features.
If others think this is a good idea we can maybe form a small team that
works on this doc and share it to be reviewed with the community (happy to
participate on this)

On Mon, May 20, 2024 at 7:02 AM Amogh Desai <am...@gmail.com>
wrote:

> I agree with Andrey too on this.
> Thanks & Regards,
> Amogh Desai
>
>
> On Fri, May 17, 2024 at 7:42 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Agreed on your points @andrey.anshin@taragol.is <
> andrey.anshin@taragol.is>
> >
> > On Fri, 17 May 2024 at 15:01, Andrey Anshin <an...@taragol.is>
> > wrote:
> >
> > > IMHO, In case if we decide to keep only Postgres support we need to
> have
> > > really powerful arguments to provide an interface which helps integrate
> > > with other DBs.
> > >
> > > In this case, we must clearly understand what the community is
> > responsible
> > > for in this case and how it can be sure that nothing is broken
> > >
> > > Especially if we take in account the Airflow has very tight
> integrations
> > > with specific Databases, requires a lot of effort to support additional
> > > ones (MS SQL case), and the DB part is not a part of the Public
> Interface
> > > of Airflow [1].
> > >
> > > So I would consider that this should be two separate decisions:
> > > 1. Keep only Postgres (vanila, not forks) as supported/tested backend
> in
> > > Production. SQLite remains as development DB.
> > > 2. Provide public interface to DB integrations between Airflow and DB
> for
> > > third parties
> > >
> > > [1]:
> > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html
> > >
> > >
> > > On Tue, 14 May 2024 at 12:15, Kaxil Naik <ka...@gmail.com> wrote:
> > >
> > > > Yeah, that works for me
> > > >
> > > > Can we have it possible to have two (or maybe three -
> > > > > like a sub-committee) co-owners of topics?
> > > >
> > > >
> > > > On Tue, 14 May 2024 at 06:15, Vikram Koka
> <vikram@astronomer.io.invalid
> > >
> > > > wrote:
> > > >
> > > > > Definitely a fast moving thread on the mailing list. I haven’t been
> > > able
> > > > to
> > > > > respond for a few days and feel very far behind already.
> > > > >
> > > > > A few comments on topics discussed the last few days:
> > > > > - Jarek, in response to your comments around being more aggressive
> > than
> > > > in
> > > > > Airflow 2 about deprecation and drops of functionality, I am very
> > > > > supportive of that stance. I completely agree that we could have
> been
> > > > more
> > > > > aggressive as part of Airflow 2.
> > > > > However, I would like to ask that as we go forward, we make sure
> that
> > > we
> > > > > have clean interfaces to be able to add support, even if we choose
> a
> > > > single
> > > > > implementation. For example, with respect to dropping MySQL
> support.
> > I
> > > > can
> > > > > understand the perspective of the project that this should be
> > > deprecated
> > > > > from an Airflow OSS perspective. However, even if the only OSS
> > > supported
> > > > DB
> > > > > is Postgres, I would like to ensure that a clean interface exists
> for
> > > > > interaction with the DB, so that other databases such as MySQL or
> > > others
> > > > > CAN be supported by a third party or at a later date.
> > > > > I realize that this may seem onerous, but I believe that it enables
> > us
> > > to
> > > > > be more flexible in the long run, rather than locking us into a
> > single
> > > DB
> > > > > implementation.
> > > > >
> > > > > - Bolke, Daniel Standish, Ash, et al on the task execution
> contract,
> > > > > definitely looking forward to this.
> > > > >
> > > > > - To those that I proposed a couple of more detailed write ups, I
> > still
> > > > > plan to do that, at the latest by early next week.
> > > > >
> > > > > Vikram
> > > > >
> > > > >
> > > > >
> > > > > On Mon, May 13, 2024 at 9:30 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > >
> > > > > > Super-excited about that.
> > > > > >
> > > > > > Question/Proposal: Can we have it possible to have two (or maybe
> > > three
> > > > -
> > > > > > like a sub-committee) co-owners of topics? I think it's a lot to
> > put
> > > on
> > > > > > one's head to "own" a topic and given circumstances/ volunteer
> time
> > > of
> > > > > > people, interruptions (and life intervening), it might be a bit
> > risky
> > > > to
> > > > > > put it on one's shoulders only.
> > > > > >
> > > > > > I know it's against the rule ("if it is owned by many, it's not
> > owned
> > > > by
> > > > > > anyone") - but I think in our case there are at least some topics
> > > that
> > > > > > could benefit from having more than one owner. Especially when we
> > > know
> > > > > and
> > > > > > trust that we can work together on some topics that we are
> > passionate
> > > > > > about. It might also encourage getting out of people's comfort
> > zones.
> > > > > >
> > > > > > For example - I'd absolutely love to volunteer to co-own the
> > > > "streamline
> > > > > > the development" with Andrey if he would be willing to of course
> :D
> > > > > (sorry
> > > > > > Andrey for "volunteering you" on that one :D) - and maybe we
> could
> > > get
> > > > > > someone else to join us.
> > > > > >
> > > > > > That might have the added benefit of being able to break with the
> > way
> > > > > > we've been doing things. If I am owning it for one - I'd likely
> > > > gravitate
> > > > > > towards past choices, but with others joining me and taking
> > decisions
> > > > > (and
> > > > > > responsibility in making sure we implement them) together, we
> could
> > > > make
> > > > > > better decisions and reduce bus factor for dev tooling/ CI in the
> > > > future.
> > > > > >
> > > > > > BTW.  Shameless promotion: tomorrow I am giving a talk about that
> > > very
> > > > > > topic (in the context of last few years not yet Airflow 3.0) at
> the
> > > NY
> > > > > > meetup hosted at Astronomer NY headquarters
> > > > > >
> https://www.meetup.com/nyc-apache-airflow-meetup/events/300017228/
> > -
> > > > so
> > > > > if
> > > > > > you are in NY or around - I think you can stil sign up :D. I am
> > also
> > > > > > getting to PyCon US in Pittsburgh next week so don't expect too
> > much
> > > > from
> > > > > > me. I will be gearing up for streamlining the development by
> > talking
> > > to
> > > > > the
> > > > > > right people and listening to the latest things and best
> practices
> > of
> > > > the
> > > > > > larger Python community :).
> > > > > >
> > > > > > J.
> > > > > >
> > > > > > On Tue, May 14, 2024 at 12:03 AM Kaxil Naik <kaxilnaik@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Thank you all, I am very happy about the discussions.
> > > > > > >
> > > > > > > The mailing list moves fast :). The main reason I recommended
> > > > starting
> > > > > > the
> > > > > > > dev calls in early June was to have some of these discussions
> on
> > > the
> > > > > > > mailing list.
> > > > > > >
> > > > > > > Since Michal already scheduled a call, let's start there to
> > discuss
> > > > > > > various ideas. For the week after that, I have created an
> Airflow
> > > > > 2-style
> > > > > > > recurring open dev calls for anyone to join, info below:
> > > > > > >
> > > > > > > *Date & Time: *Recurring every 2 weeks on Thursday at* 4pm BST
> > *( 3
> > > > PM
> > > > > > > GMT/UTC | 11 AM EST | 8 AM PST); starting* May 30, 2024 04:00
> PM
> > > BST*
> > > > > and
> > > > > > > then
> > > > > > > *One-time registration Link*:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> > > > > > > *Add to your calendar*:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add
> > > > > > >
> > > > > > > I will post the meeting notes on the dev mailing list as well
> as
> > > > > > Confluence
> > > > > > > for archival purposes (example
> > > > > > > <
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes
> > > > >).
> > > > > > >
> > > > > > > Once we discuss various proposals next week, I recommend that
> for
> > > > each
> > > > > > > "workstream", we have an owner who would want to lead that
> > > > workstream.
> > > > > > For
> > > > > > > items, that does not have an owner we can put those into
> Airflow
> > 3
> > > > Meta
> > > > > > > issue <https://github.com/apache/airflow/issues/39593> or
> > > cross-link
> > > > > > over
> > > > > > > there so someone in the community can take it on. If we don't
> > have
> > > an
> > > > > > owner
> > > > > > > who will commit to working on it, we park that item until we
> find
> > > the
> > > > > > > owner.
> > > > > > >
> > > > > > > At the end of each call, I would solicit ideas for the agenda
> for
> > > the
> > > > > > next
> > > > > > > call and propose it to the broader group on the mailing list.
> > > > > > >
> > > > > > > Some of the items that should be discussed in the upcoming
> calls
> > > IMO:
> > > > > > >
> > > > > > >    - Agreeing on Principles
> > > > > > >
> > > > > > >    Based on the discussions, some potential items (all up for
> > > debate)
> > > > > > >       - Considering Airflow 3.0 for early adopters and*
> breaking
> > > (and
> > > > > > >       removing) things for AF 3.0*. Things can be re-added as
> > > needed
> > > > in
> > > > > > >       upcoming minor releases
> > > > > > >       - Optimize to get *foundational pieces in* and not "let
> > > perfect
> > > > > be
> > > > > > >       the enemy of good"
> > > > > > >       - Working on features that solidify Airflow as the*
> modern
> > > > > > >       Orchestrator* that also has state of the art *support for
> > > Data,
> > > > > AI
> > > > > > &
> > > > > > >       ML workloads*. This includes scalability & performance
> > > > discussion
> > > > > > >       - Set up the codebase for the next 5 years. This
> > encompasses
> > > > all
> > > > > > the
> > > > > > >       things we are discussing e.g removing MySQL to reduce the
> > > test
> > > > > > > matrix,
> > > > > > >       simplifying things architecturally, consolidating
> > > serialization
> > > > > > > methods, etc
> > > > > > >
> > > > > > >       - Workstream & Stream Owners
> > > > > > >    - Airflow 2 support policy including scope (feature vs bug
> > > fixes +
> > > > > > >    security only) & support period
> > > > > > >    - Separate discussions for each big workstream including one
> > for
> > > > > items
> > > > > > >    to remove & refactor (e.g dropping MySQL)
> > > > > > >    - Discussion to streamline the development of Airflow 3
> > > > > > >       - Separating dev for Providers & Airflow (something Jarek
> > > > already
> > > > > > >       kick-started), and
> > > > > > >       - Separate branch for Airflow 2
> > > > > > >       - CI changes for the above
> > > > > > >    - Finalize Scope + Timelines
> > > > > > >    - Migration Utilities
> > > > > > >    - Progress check-ins
> > > > > > >
> > > > > > > Looking forward to the exciting months ahead.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Kaxil
> > > > > > >
> > > > > > > On Mon, 13 May 2024 at 21:40, Bolke de Bruin <
> bdbruin@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Declaring connections prior to task execution was already
> > > proposed
> > > > in
> > > > > > > AIP-1
> > > > > > > > :-). At that time, I had in mind to communicate over IPC to
> the
> > > > task
> > > > > > the
> > > > > > > > required settings. Registration could then happen with a
> > > manifest.
> > > > > > Maybe
> > > > > > > > during DAG serialization this could be obtained
> unobtrusively?
> > > The
> > > > > > > benefit
> > > > > > > > is that tasks become truly atomic or independent from Airflow
> > as
> > > > long
> > > > > > as
> > > > > > > > they communicate their exit codes (success, failed, and I
> think
> > > Ash
> > > > > > had a
> > > > > > > > couple of others in mind - the fewer the better).
> > > > > > > >
> > > > > > > > If you want two-way communication, maybe for variables as
> they
> > > can
> > > > > > change
> > > > > > > > during scheduling, this can happen with AIP-44. Although, I'd
> > > > prefer
> > > > > it
> > > > > > > to
> > > > > > > > happen with the *executor* rather than some centralized
> > service.
> > > If
> > > > > the
> > > > > > > > executor is used, IPC is the logical choice. The benefit of
> > this
> > > is
> > > > > > that
> > > > > > > > you have better resiliency and you can start to think about
> no
> > > > > downtime
> > > > > > > > upgrades
> > > > > > > >
> > > > > > > > So I hope Ash takes this to 2024 :-).
> > > > > > > >
> > > > > > > > B.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <
> > ash@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > > That would require some mechanism of declaring prior to
> > task
> > > > > > > execution
> > > > > > > > > what connections would be used
> > > > > > > > >
> > > > > > > > > That’s exactly what I’m proposing in the proposal doc I’m
> > > working
> > > > > on
> > > > > > > > (It’s
> > > > > > > > > part of also overhauling and re-designing the “Task
> Execution
> > > > > > > interface”
> > > > > > > > > that also gives us the ability to nicely have support for
> > > running
> > > > > > tasks
> > > > > > > > in
> > > > > > > > > other languages — much more than just BashOperator)
> > > > > > > > >
> > > > > > > > > This is a bit of a fundamental shift in thinking about task
> > > > > execution
> > > > > > > in
> > > > > > > > > Airflow, but I think it gives us some really nice
> properties
> > > that
> > > > > the
> > > > > > > > > project is currently missing.
> > > > > > > > >
> > > > > > > > > Tl;dr; lets discuss this in my doc when it comes our (next
> > week
> > > > > most
> > > > > > > > > likely) please :)
> > > > > > > > >
> > > > > > > > > -ash
> > > > > > > > >
> > > > > > > > > > On 13 May 2024, at 18:15, Daniel Standish
> > > > > > > > > <da...@astronomer.io.INVALID> wrote:
> > > > > > > > > >
> > > > > > > > > > re
> > > > > > > > > >
> > > > > > > > > > As tasks require connection access, I assume connection
> > data
> > > > will
> > > > > > > > somehow
> > > > > > > > > >> be passed as part of the
> > > > > > > > > >> metadata to task execution - whether it's part of the
> > > executor
> > > > > > > > protocol
> > > > > > > > > or
> > > > > > > > > >> in some other way (I'm
> > > > > > > > > >> not an expert on that part of Airflow). Then, provided
> > it's
> > > > > > > accessible
> > > > > > > > > as
> > > > > > > > > >> part of some execution
> > > > > > > > > >> context, and not only passed to the task's execute
> method,
> > > > > > > OpenLineage
> > > > > > > > > >> could utilize it.
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > > It's not strictly necessary that connection info be
> passed
> > > "as
> > > > > part
> > > > > > > of
> > > > > > > > > task
> > > > > > > > > > matadata".  That would require some mechanism of
> declaring
> > > > prior
> > > > > to
> > > > > > > > task
> > > > > > > > > > execution what connections would be used.  This is a
> > thought
> > > > that
> > > > > > has
> > > > > > > > > come
> > > > > > > > > > up when thinking about execution of non-python tasks.
> But
> > > it's
> > > > > not
> > > > > > > > > > required from a technical perspective by AIP-44 because
> the
> > > > > > > > > > `get_connection` function can be made to be an RPC call
> so
> > a
> > > > task
> > > > > > > could
> > > > > > > > > > continue to retrieve connections at runtime.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > --
> > > > > > > > Bolke de Bruin
> > > > > > > > bdbruin@gmail.com
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Amogh Desai <am...@gmail.com>]

I agree with Andrey too on this.
Thanks & Regards,
Amogh Desai


On Fri, May 17, 2024 at 7:42 PM Kaxil Naik <ka...@gmail.com> wrote:

> Agreed on your points @andrey.anshin@taragol.is <an...@taragol.is>
>
> On Fri, 17 May 2024 at 15:01, Andrey Anshin <an...@taragol.is>
> wrote:
>
> > IMHO, In case if we decide to keep only Postgres support we need to have
> > really powerful arguments to provide an interface which helps integrate
> > with other DBs.
> >
> > In this case, we must clearly understand what the community is
> responsible
> > for in this case and how it can be sure that nothing is broken
> >
> > Especially if we take in account the Airflow has very tight integrations
> > with specific Databases, requires a lot of effort to support additional
> > ones (MS SQL case), and the DB part is not a part of the Public Interface
> > of Airflow [1].
> >
> > So I would consider that this should be two separate decisions:
> > 1. Keep only Postgres (vanila, not forks) as supported/tested backend in
> > Production. SQLite remains as development DB.
> > 2. Provide public interface to DB integrations between Airflow and DB for
> > third parties
> >
> > [1]:
> >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html
> >
> >
> > On Tue, 14 May 2024 at 12:15, Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Yeah, that works for me
> > >
> > > Can we have it possible to have two (or maybe three -
> > > > like a sub-committee) co-owners of topics?
> > >
> > >
> > > On Tue, 14 May 2024 at 06:15, Vikram Koka <vikram@astronomer.io.invalid
> >
> > > wrote:
> > >
> > > > Definitely a fast moving thread on the mailing list. I haven’t been
> > able
> > > to
> > > > respond for a few days and feel very far behind already.
> > > >
> > > > A few comments on topics discussed the last few days:
> > > > - Jarek, in response to your comments around being more aggressive
> than
> > > in
> > > > Airflow 2 about deprecation and drops of functionality, I am very
> > > > supportive of that stance. I completely agree that we could have been
> > > more
> > > > aggressive as part of Airflow 2.
> > > > However, I would like to ask that as we go forward, we make sure that
> > we
> > > > have clean interfaces to be able to add support, even if we choose a
> > > single
> > > > implementation. For example, with respect to dropping MySQL support.
> I
> > > can
> > > > understand the perspective of the project that this should be
> > deprecated
> > > > from an Airflow OSS perspective. However, even if the only OSS
> > supported
> > > DB
> > > > is Postgres, I would like to ensure that a clean interface exists for
> > > > interaction with the DB, so that other databases such as MySQL or
> > others
> > > > CAN be supported by a third party or at a later date.
> > > > I realize that this may seem onerous, but I believe that it enables
> us
> > to
> > > > be more flexible in the long run, rather than locking us into a
> single
> > DB
> > > > implementation.
> > > >
> > > > - Bolke, Daniel Standish, Ash, et al on the task execution contract,
> > > > definitely looking forward to this.
> > > >
> > > > - To those that I proposed a couple of more detailed write ups, I
> still
> > > > plan to do that, at the latest by early next week.
> > > >
> > > > Vikram
> > > >
> > > >
> > > >
> > > > On Mon, May 13, 2024 at 9:30 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > > > Super-excited about that.
> > > > >
> > > > > Question/Proposal: Can we have it possible to have two (or maybe
> > three
> > > -
> > > > > like a sub-committee) co-owners of topics? I think it's a lot to
> put
> > on
> > > > > one's head to "own" a topic and given circumstances/ volunteer time
> > of
> > > > > people, interruptions (and life intervening), it might be a bit
> risky
> > > to
> > > > > put it on one's shoulders only.
> > > > >
> > > > > I know it's against the rule ("if it is owned by many, it's not
> owned
> > > by
> > > > > anyone") - but I think in our case there are at least some topics
> > that
> > > > > could benefit from having more than one owner. Especially when we
> > know
> > > > and
> > > > > trust that we can work together on some topics that we are
> passionate
> > > > > about. It might also encourage getting out of people's comfort
> zones.
> > > > >
> > > > > For example - I'd absolutely love to volunteer to co-own the
> > > "streamline
> > > > > the development" with Andrey if he would be willing to of course :D
> > > > (sorry
> > > > > Andrey for "volunteering you" on that one :D) - and maybe we could
> > get
> > > > > someone else to join us.
> > > > >
> > > > > That might have the added benefit of being able to break with the
> way
> > > > > we've been doing things. If I am owning it for one - I'd likely
> > > gravitate
> > > > > towards past choices, but with others joining me and taking
> decisions
> > > > (and
> > > > > responsibility in making sure we implement them) together, we could
> > > make
> > > > > better decisions and reduce bus factor for dev tooling/ CI in the
> > > future.
> > > > >
> > > > > BTW.  Shameless promotion: tomorrow I am giving a talk about that
> > very
> > > > > topic (in the context of last few years not yet Airflow 3.0) at the
> > NY
> > > > > meetup hosted at Astronomer NY headquarters
> > > > > https://www.meetup.com/nyc-apache-airflow-meetup/events/300017228/
> -
> > > so
> > > > if
> > > > > you are in NY or around - I think you can stil sign up :D. I am
> also
> > > > > getting to PyCon US in Pittsburgh next week so don't expect too
> much
> > > from
> > > > > me. I will be gearing up for streamlining the development by
> talking
> > to
> > > > the
> > > > > right people and listening to the latest things and best practices
> of
> > > the
> > > > > larger Python community :).
> > > > >
> > > > > J.
> > > > >
> > > > > On Tue, May 14, 2024 at 12:03 AM Kaxil Naik <ka...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thank you all, I am very happy about the discussions.
> > > > > >
> > > > > > The mailing list moves fast :). The main reason I recommended
> > > starting
> > > > > the
> > > > > > dev calls in early June was to have some of these discussions on
> > the
> > > > > > mailing list.
> > > > > >
> > > > > > Since Michal already scheduled a call, let's start there to
> discuss
> > > > > > various ideas. For the week after that, I have created an Airflow
> > > > 2-style
> > > > > > recurring open dev calls for anyone to join, info below:
> > > > > >
> > > > > > *Date & Time: *Recurring every 2 weeks on Thursday at* 4pm BST
> *( 3
> > > PM
> > > > > > GMT/UTC | 11 AM EST | 8 AM PST); starting* May 30, 2024 04:00 PM
> > BST*
> > > > and
> > > > > > then
> > > > > > *One-time registration Link*:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> > > > > > *Add to your calendar*:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add
> > > > > >
> > > > > > I will post the meeting notes on the dev mailing list as well as
> > > > > Confluence
> > > > > > for archival purposes (example
> > > > > > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes
> > > >).
> > > > > >
> > > > > > Once we discuss various proposals next week, I recommend that for
> > > each
> > > > > > "workstream", we have an owner who would want to lead that
> > > workstream.
> > > > > For
> > > > > > items, that does not have an owner we can put those into Airflow
> 3
> > > Meta
> > > > > > issue <https://github.com/apache/airflow/issues/39593> or
> > cross-link
> > > > > over
> > > > > > there so someone in the community can take it on. If we don't
> have
> > an
> > > > > owner
> > > > > > who will commit to working on it, we park that item until we find
> > the
> > > > > > owner.
> > > > > >
> > > > > > At the end of each call, I would solicit ideas for the agenda for
> > the
> > > > > next
> > > > > > call and propose it to the broader group on the mailing list.
> > > > > >
> > > > > > Some of the items that should be discussed in the upcoming calls
> > IMO:
> > > > > >
> > > > > >    - Agreeing on Principles
> > > > > >
> > > > > >    Based on the discussions, some potential items (all up for
> > debate)
> > > > > >       - Considering Airflow 3.0 for early adopters and* breaking
> > (and
> > > > > >       removing) things for AF 3.0*. Things can be re-added as
> > needed
> > > in
> > > > > >       upcoming minor releases
> > > > > >       - Optimize to get *foundational pieces in* and not "let
> > perfect
> > > > be
> > > > > >       the enemy of good"
> > > > > >       - Working on features that solidify Airflow as the* modern
> > > > > >       Orchestrator* that also has state of the art *support for
> > Data,
> > > > AI
> > > > > &
> > > > > >       ML workloads*. This includes scalability & performance
> > > discussion
> > > > > >       - Set up the codebase for the next 5 years. This
> encompasses
> > > all
> > > > > the
> > > > > >       things we are discussing e.g removing MySQL to reduce the
> > test
> > > > > > matrix,
> > > > > >       simplifying things architecturally, consolidating
> > serialization
> > > > > > methods, etc
> > > > > >
> > > > > >       - Workstream & Stream Owners
> > > > > >    - Airflow 2 support policy including scope (feature vs bug
> > fixes +
> > > > > >    security only) & support period
> > > > > >    - Separate discussions for each big workstream including one
> for
> > > > items
> > > > > >    to remove & refactor (e.g dropping MySQL)
> > > > > >    - Discussion to streamline the development of Airflow 3
> > > > > >       - Separating dev for Providers & Airflow (something Jarek
> > > already
> > > > > >       kick-started), and
> > > > > >       - Separate branch for Airflow 2
> > > > > >       - CI changes for the above
> > > > > >    - Finalize Scope + Timelines
> > > > > >    - Migration Utilities
> > > > > >    - Progress check-ins
> > > > > >
> > > > > > Looking forward to the exciting months ahead.
> > > > > >
> > > > > > Regards,
> > > > > > Kaxil
> > > > > >
> > > > > > On Mon, 13 May 2024 at 21:40, Bolke de Bruin <bd...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Declaring connections prior to task execution was already
> > proposed
> > > in
> > > > > > AIP-1
> > > > > > > :-). At that time, I had in mind to communicate over IPC to the
> > > task
> > > > > the
> > > > > > > required settings. Registration could then happen with a
> > manifest.
> > > > > Maybe
> > > > > > > during DAG serialization this could be obtained unobtrusively?
> > The
> > > > > > benefit
> > > > > > > is that tasks become truly atomic or independent from Airflow
> as
> > > long
> > > > > as
> > > > > > > they communicate their exit codes (success, failed, and I think
> > Ash
> > > > > had a
> > > > > > > couple of others in mind - the fewer the better).
> > > > > > >
> > > > > > > If you want two-way communication, maybe for variables as they
> > can
> > > > > change
> > > > > > > during scheduling, this can happen with AIP-44. Although, I'd
> > > prefer
> > > > it
> > > > > > to
> > > > > > > happen with the *executor* rather than some centralized
> service.
> > If
> > > > the
> > > > > > > executor is used, IPC is the logical choice. The benefit of
> this
> > is
> > > > > that
> > > > > > > you have better resiliency and you can start to think about no
> > > > downtime
> > > > > > > upgrades
> > > > > > >
> > > > > > > So I hope Ash takes this to 2024 :-).
> > > > > > >
> > > > > > > B.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <
> ash@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > > That would require some mechanism of declaring prior to
> task
> > > > > > execution
> > > > > > > > what connections would be used
> > > > > > > >
> > > > > > > > That’s exactly what I’m proposing in the proposal doc I’m
> > working
> > > > on
> > > > > > > (It’s
> > > > > > > > part of also overhauling and re-designing the “Task Execution
> > > > > > interface”
> > > > > > > > that also gives us the ability to nicely have support for
> > running
> > > > > tasks
> > > > > > > in
> > > > > > > > other languages — much more than just BashOperator)
> > > > > > > >
> > > > > > > > This is a bit of a fundamental shift in thinking about task
> > > > execution
> > > > > > in
> > > > > > > > Airflow, but I think it gives us some really nice properties
> > that
> > > > the
> > > > > > > > project is currently missing.
> > > > > > > >
> > > > > > > > Tl;dr; lets discuss this in my doc when it comes our (next
> week
> > > > most
> > > > > > > > likely) please :)
> > > > > > > >
> > > > > > > > -ash
> > > > > > > >
> > > > > > > > > On 13 May 2024, at 18:15, Daniel Standish
> > > > > > > > <da...@astronomer.io.INVALID> wrote:
> > > > > > > > >
> > > > > > > > > re
> > > > > > > > >
> > > > > > > > > As tasks require connection access, I assume connection
> data
> > > will
> > > > > > > somehow
> > > > > > > > >> be passed as part of the
> > > > > > > > >> metadata to task execution - whether it's part of the
> > executor
> > > > > > > protocol
> > > > > > > > or
> > > > > > > > >> in some other way (I'm
> > > > > > > > >> not an expert on that part of Airflow). Then, provided
> it's
> > > > > > accessible
> > > > > > > > as
> > > > > > > > >> part of some execution
> > > > > > > > >> context, and not only passed to the task's execute method,
> > > > > > OpenLineage
> > > > > > > > >> could utilize it.
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > > It's not strictly necessary that connection info be passed
> > "as
> > > > part
> > > > > > of
> > > > > > > > task
> > > > > > > > > matadata".  That would require some mechanism of declaring
> > > prior
> > > > to
> > > > > > > task
> > > > > > > > > execution what connections would be used.  This is a
> thought
> > > that
> > > > > has
> > > > > > > > come
> > > > > > > > > up when thinking about execution of non-python tasks.  But
> > it's
> > > > not
> > > > > > > > > required from a technical perspective by AIP-44 because the
> > > > > > > > > `get_connection` function can be made to be an RPC call so
> a
> > > task
> > > > > > could
> > > > > > > > > continue to retrieve connections at runtime.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > --
> > > > > > > Bolke de Bruin
> > > > > > > bdbruin@gmail.com
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Kaxil Naik <ka...@gmail.com>]

Agreed on your points @andrey.anshin@taragol.is <an...@taragol.is>

On Fri, 17 May 2024 at 15:01, Andrey Anshin <an...@taragol.is>
wrote:

> IMHO, In case if we decide to keep only Postgres support we need to have
> really powerful arguments to provide an interface which helps integrate
> with other DBs.
>
> In this case, we must clearly understand what the community is responsible
> for in this case and how it can be sure that nothing is broken
>
> Especially if we take in account the Airflow has very tight integrations
> with specific Databases, requires a lot of effort to support additional
> ones (MS SQL case), and the DB part is not a part of the Public Interface
> of Airflow [1].
>
> So I would consider that this should be two separate decisions:
> 1. Keep only Postgres (vanila, not forks) as supported/tested backend in
> Production. SQLite remains as development DB.
> 2. Provide public interface to DB integrations between Airflow and DB for
> third parties
>
> [1]:
>
> https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html
>
>
> On Tue, 14 May 2024 at 12:15, Kaxil Naik <ka...@gmail.com> wrote:
>
> > Yeah, that works for me
> >
> > Can we have it possible to have two (or maybe three -
> > > like a sub-committee) co-owners of topics?
> >
> >
> > On Tue, 14 May 2024 at 06:15, Vikram Koka <vi...@astronomer.io.invalid>
> > wrote:
> >
> > > Definitely a fast moving thread on the mailing list. I haven’t been
> able
> > to
> > > respond for a few days and feel very far behind already.
> > >
> > > A few comments on topics discussed the last few days:
> > > - Jarek, in response to your comments around being more aggressive than
> > in
> > > Airflow 2 about deprecation and drops of functionality, I am very
> > > supportive of that stance. I completely agree that we could have been
> > more
> > > aggressive as part of Airflow 2.
> > > However, I would like to ask that as we go forward, we make sure that
> we
> > > have clean interfaces to be able to add support, even if we choose a
> > single
> > > implementation. For example, with respect to dropping MySQL support. I
> > can
> > > understand the perspective of the project that this should be
> deprecated
> > > from an Airflow OSS perspective. However, even if the only OSS
> supported
> > DB
> > > is Postgres, I would like to ensure that a clean interface exists for
> > > interaction with the DB, so that other databases such as MySQL or
> others
> > > CAN be supported by a third party or at a later date.
> > > I realize that this may seem onerous, but I believe that it enables us
> to
> > > be more flexible in the long run, rather than locking us into a single
> DB
> > > implementation.
> > >
> > > - Bolke, Daniel Standish, Ash, et al on the task execution contract,
> > > definitely looking forward to this.
> > >
> > > - To those that I proposed a couple of more detailed write ups, I still
> > > plan to do that, at the latest by early next week.
> > >
> > > Vikram
> > >
> > >
> > >
> > > On Mon, May 13, 2024 at 9:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Super-excited about that.
> > > >
> > > > Question/Proposal: Can we have it possible to have two (or maybe
> three
> > -
> > > > like a sub-committee) co-owners of topics? I think it's a lot to put
> on
> > > > one's head to "own" a topic and given circumstances/ volunteer time
> of
> > > > people, interruptions (and life intervening), it might be a bit risky
> > to
> > > > put it on one's shoulders only.
> > > >
> > > > I know it's against the rule ("if it is owned by many, it's not owned
> > by
> > > > anyone") - but I think in our case there are at least some topics
> that
> > > > could benefit from having more than one owner. Especially when we
> know
> > > and
> > > > trust that we can work together on some topics that we are passionate
> > > > about. It might also encourage getting out of people's comfort zones.
> > > >
> > > > For example - I'd absolutely love to volunteer to co-own the
> > "streamline
> > > > the development" with Andrey if he would be willing to of course :D
> > > (sorry
> > > > Andrey for "volunteering you" on that one :D) - and maybe we could
> get
> > > > someone else to join us.
> > > >
> > > > That might have the added benefit of being able to break with the way
> > > > we've been doing things. If I am owning it for one - I'd likely
> > gravitate
> > > > towards past choices, but with others joining me and taking decisions
> > > (and
> > > > responsibility in making sure we implement them) together, we could
> > make
> > > > better decisions and reduce bus factor for dev tooling/ CI in the
> > future.
> > > >
> > > > BTW.  Shameless promotion: tomorrow I am giving a talk about that
> very
> > > > topic (in the context of last few years not yet Airflow 3.0) at the
> NY
> > > > meetup hosted at Astronomer NY headquarters
> > > > https://www.meetup.com/nyc-apache-airflow-meetup/events/300017228/ -
> > so
> > > if
> > > > you are in NY or around - I think you can stil sign up :D. I am also
> > > > getting to PyCon US in Pittsburgh next week so don't expect too much
> > from
> > > > me. I will be gearing up for streamlining the development by talking
> to
> > > the
> > > > right people and listening to the latest things and best practices of
> > the
> > > > larger Python community :).
> > > >
> > > > J.
> > > >
> > > > On Tue, May 14, 2024 at 12:03 AM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > > >
> > > > > Thank you all, I am very happy about the discussions.
> > > > >
> > > > > The mailing list moves fast :). The main reason I recommended
> > starting
> > > > the
> > > > > dev calls in early June was to have some of these discussions on
> the
> > > > > mailing list.
> > > > >
> > > > > Since Michal already scheduled a call, let's start there to discuss
> > > > > various ideas. For the week after that, I have created an Airflow
> > > 2-style
> > > > > recurring open dev calls for anyone to join, info below:
> > > > >
> > > > > *Date & Time: *Recurring every 2 weeks on Thursday at* 4pm BST *( 3
> > PM
> > > > > GMT/UTC | 11 AM EST | 8 AM PST); starting* May 30, 2024 04:00 PM
> BST*
> > > and
> > > > > then
> > > > > *One-time registration Link*:
> > > > >
> > > > >
> > > >
> > >
> >
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> > > > > *Add to your calendar*:
> > > > >
> > > > >
> > > >
> > >
> >
> https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add
> > > > >
> > > > > I will post the meeting notes on the dev mailing list as well as
> > > > Confluence
> > > > > for archival purposes (example
> > > > > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes
> > >).
> > > > >
> > > > > Once we discuss various proposals next week, I recommend that for
> > each
> > > > > "workstream", we have an owner who would want to lead that
> > workstream.
> > > > For
> > > > > items, that does not have an owner we can put those into Airflow 3
> > Meta
> > > > > issue <https://github.com/apache/airflow/issues/39593> or
> cross-link
> > > > over
> > > > > there so someone in the community can take it on. If we don't have
> an
> > > > owner
> > > > > who will commit to working on it, we park that item until we find
> the
> > > > > owner.
> > > > >
> > > > > At the end of each call, I would solicit ideas for the agenda for
> the
> > > > next
> > > > > call and propose it to the broader group on the mailing list.
> > > > >
> > > > > Some of the items that should be discussed in the upcoming calls
> IMO:
> > > > >
> > > > >    - Agreeing on Principles
> > > > >
> > > > >    Based on the discussions, some potential items (all up for
> debate)
> > > > >       - Considering Airflow 3.0 for early adopters and* breaking
> (and
> > > > >       removing) things for AF 3.0*. Things can be re-added as
> needed
> > in
> > > > >       upcoming minor releases
> > > > >       - Optimize to get *foundational pieces in* and not "let
> perfect
> > > be
> > > > >       the enemy of good"
> > > > >       - Working on features that solidify Airflow as the* modern
> > > > >       Orchestrator* that also has state of the art *support for
> Data,
> > > AI
> > > > &
> > > > >       ML workloads*. This includes scalability & performance
> > discussion
> > > > >       - Set up the codebase for the next 5 years. This encompasses
> > all
> > > > the
> > > > >       things we are discussing e.g removing MySQL to reduce the
> test
> > > > > matrix,
> > > > >       simplifying things architecturally, consolidating
> serialization
> > > > > methods, etc
> > > > >
> > > > >       - Workstream & Stream Owners
> > > > >    - Airflow 2 support policy including scope (feature vs bug
> fixes +
> > > > >    security only) & support period
> > > > >    - Separate discussions for each big workstream including one for
> > > items
> > > > >    to remove & refactor (e.g dropping MySQL)
> > > > >    - Discussion to streamline the development of Airflow 3
> > > > >       - Separating dev for Providers & Airflow (something Jarek
> > already
> > > > >       kick-started), and
> > > > >       - Separate branch for Airflow 2
> > > > >       - CI changes for the above
> > > > >    - Finalize Scope + Timelines
> > > > >    - Migration Utilities
> > > > >    - Progress check-ins
> > > > >
> > > > > Looking forward to the exciting months ahead.
> > > > >
> > > > > Regards,
> > > > > Kaxil
> > > > >
> > > > > On Mon, 13 May 2024 at 21:40, Bolke de Bruin <bd...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Declaring connections prior to task execution was already
> proposed
> > in
> > > > > AIP-1
> > > > > > :-). At that time, I had in mind to communicate over IPC to the
> > task
> > > > the
> > > > > > required settings. Registration could then happen with a
> manifest.
> > > > Maybe
> > > > > > during DAG serialization this could be obtained unobtrusively?
> The
> > > > > benefit
> > > > > > is that tasks become truly atomic or independent from Airflow as
> > long
> > > > as
> > > > > > they communicate their exit codes (success, failed, and I think
> Ash
> > > > had a
> > > > > > couple of others in mind - the fewer the better).
> > > > > >
> > > > > > If you want two-way communication, maybe for variables as they
> can
> > > > change
> > > > > > during scheduling, this can happen with AIP-44. Although, I'd
> > prefer
> > > it
> > > > > to
> > > > > > happen with the *executor* rather than some centralized service.
> If
> > > the
> > > > > > executor is used, IPC is the logical choice. The benefit of this
> is
> > > > that
> > > > > > you have better resiliency and you can start to think about no
> > > downtime
> > > > > > upgrades
> > > > > >
> > > > > > So I hope Ash takes this to 2024 :-).
> > > > > >
> > > > > > B.
> > > > > >
> > > > > >
> > > > > > On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <as...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > > That would require some mechanism of declaring prior to task
> > > > > execution
> > > > > > > what connections would be used
> > > > > > >
> > > > > > > That’s exactly what I’m proposing in the proposal doc I’m
> working
> > > on
> > > > > > (It’s
> > > > > > > part of also overhauling and re-designing the “Task Execution
> > > > > interface”
> > > > > > > that also gives us the ability to nicely have support for
> running
> > > > tasks
> > > > > > in
> > > > > > > other languages — much more than just BashOperator)
> > > > > > >
> > > > > > > This is a bit of a fundamental shift in thinking about task
> > > execution
> > > > > in
> > > > > > > Airflow, but I think it gives us some really nice properties
> that
> > > the
> > > > > > > project is currently missing.
> > > > > > >
> > > > > > > Tl;dr; lets discuss this in my doc when it comes our (next week
> > > most
> > > > > > > likely) please :)
> > > > > > >
> > > > > > > -ash
> > > > > > >
> > > > > > > > On 13 May 2024, at 18:15, Daniel Standish
> > > > > > > <da...@astronomer.io.INVALID> wrote:
> > > > > > > >
> > > > > > > > re
> > > > > > > >
> > > > > > > > As tasks require connection access, I assume connection data
> > will
> > > > > > somehow
> > > > > > > >> be passed as part of the
> > > > > > > >> metadata to task execution - whether it's part of the
> executor
> > > > > > protocol
> > > > > > > or
> > > > > > > >> in some other way (I'm
> > > > > > > >> not an expert on that part of Airflow). Then, provided it's
> > > > > accessible
> > > > > > > as
> > > > > > > >> part of some execution
> > > > > > > >> context, and not only passed to the task's execute method,
> > > > > OpenLineage
> > > > > > > >> could utilize it.
> > > > > > > >>
> > > > > > > >
> > > > > > > > It's not strictly necessary that connection info be passed
> "as
> > > part
> > > > > of
> > > > > > > task
> > > > > > > > matadata".  That would require some mechanism of declaring
> > prior
> > > to
> > > > > > task
> > > > > > > > execution what connections would be used.  This is a thought
> > that
> > > > has
> > > > > > > come
> > > > > > > > up when thinking about execution of non-python tasks.  But
> it's
> > > not
> > > > > > > > required from a technical perspective by AIP-44 because the
> > > > > > > > `get_connection` function can be made to be an RPC call so a
> > task
> > > > > could
> > > > > > > > continue to retrieve connections at runtime.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > --
> > > > > > Bolke de Bruin
> > > > > > bdbruin@gmail.com
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Andrey Anshin <an...@taragol.is>]

IMHO, In case if we decide to keep only Postgres support we need to have
really powerful arguments to provide an interface which helps integrate
with other DBs.

In this case, we must clearly understand what the community is responsible
for in this case and how it can be sure that nothing is broken

Especially if we take in account the Airflow has very tight integrations
with specific Databases, requires a lot of effort to support additional
ones (MS SQL case), and the DB part is not a part of the Public Interface
of Airflow [1].

So I would consider that this should be two separate decisions:
1. Keep only Postgres (vanila, not forks) as supported/tested backend in
Production. SQLite remains as development DB.
2. Provide public interface to DB integrations between Airflow and DB for
third parties

[1]:
https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html


On Tue, 14 May 2024 at 12:15, Kaxil Naik <ka...@gmail.com> wrote:

> Yeah, that works for me
>
> Can we have it possible to have two (or maybe three -
> > like a sub-committee) co-owners of topics?
>
>
> On Tue, 14 May 2024 at 06:15, Vikram Koka <vi...@astronomer.io.invalid>
> wrote:
>
> > Definitely a fast moving thread on the mailing list. I haven’t been able
> to
> > respond for a few days and feel very far behind already.
> >
> > A few comments on topics discussed the last few days:
> > - Jarek, in response to your comments around being more aggressive than
> in
> > Airflow 2 about deprecation and drops of functionality, I am very
> > supportive of that stance. I completely agree that we could have been
> more
> > aggressive as part of Airflow 2.
> > However, I would like to ask that as we go forward, we make sure that we
> > have clean interfaces to be able to add support, even if we choose a
> single
> > implementation. For example, with respect to dropping MySQL support. I
> can
> > understand the perspective of the project that this should be deprecated
> > from an Airflow OSS perspective. However, even if the only OSS supported
> DB
> > is Postgres, I would like to ensure that a clean interface exists for
> > interaction with the DB, so that other databases such as MySQL or others
> > CAN be supported by a third party or at a later date.
> > I realize that this may seem onerous, but I believe that it enables us to
> > be more flexible in the long run, rather than locking us into a single DB
> > implementation.
> >
> > - Bolke, Daniel Standish, Ash, et al on the task execution contract,
> > definitely looking forward to this.
> >
> > - To those that I proposed a couple of more detailed write ups, I still
> > plan to do that, at the latest by early next week.
> >
> > Vikram
> >
> >
> >
> > On Mon, May 13, 2024 at 9:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Super-excited about that.
> > >
> > > Question/Proposal: Can we have it possible to have two (or maybe three
> -
> > > like a sub-committee) co-owners of topics? I think it's a lot to put on
> > > one's head to "own" a topic and given circumstances/ volunteer time of
> > > people, interruptions (and life intervening), it might be a bit risky
> to
> > > put it on one's shoulders only.
> > >
> > > I know it's against the rule ("if it is owned by many, it's not owned
> by
> > > anyone") - but I think in our case there are at least some topics that
> > > could benefit from having more than one owner. Especially when we know
> > and
> > > trust that we can work together on some topics that we are passionate
> > > about. It might also encourage getting out of people's comfort zones.
> > >
> > > For example - I'd absolutely love to volunteer to co-own the
> "streamline
> > > the development" with Andrey if he would be willing to of course :D
> > (sorry
> > > Andrey for "volunteering you" on that one :D) - and maybe we could get
> > > someone else to join us.
> > >
> > > That might have the added benefit of being able to break with the way
> > > we've been doing things. If I am owning it for one - I'd likely
> gravitate
> > > towards past choices, but with others joining me and taking decisions
> > (and
> > > responsibility in making sure we implement them) together, we could
> make
> > > better decisions and reduce bus factor for dev tooling/ CI in the
> future.
> > >
> > > BTW.  Shameless promotion: tomorrow I am giving a talk about that very
> > > topic (in the context of last few years not yet Airflow 3.0) at the NY
> > > meetup hosted at Astronomer NY headquarters
> > > https://www.meetup.com/nyc-apache-airflow-meetup/events/300017228/ -
> so
> > if
> > > you are in NY or around - I think you can stil sign up :D. I am also
> > > getting to PyCon US in Pittsburgh next week so don't expect too much
> from
> > > me. I will be gearing up for streamlining the development by talking to
> > the
> > > right people and listening to the latest things and best practices of
> the
> > > larger Python community :).
> > >
> > > J.
> > >
> > > On Tue, May 14, 2024 at 12:03 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > > > Thank you all, I am very happy about the discussions.
> > > >
> > > > The mailing list moves fast :). The main reason I recommended
> starting
> > > the
> > > > dev calls in early June was to have some of these discussions on the
> > > > mailing list.
> > > >
> > > > Since Michal already scheduled a call, let's start there to discuss
> > > > various ideas. For the week after that, I have created an Airflow
> > 2-style
> > > > recurring open dev calls for anyone to join, info below:
> > > >
> > > > *Date & Time: *Recurring every 2 weeks on Thursday at* 4pm BST *( 3
> PM
> > > > GMT/UTC | 11 AM EST | 8 AM PST); starting* May 30, 2024 04:00 PM BST*
> > and
> > > > then
> > > > *One-time registration Link*:
> > > >
> > > >
> > >
> >
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> > > > *Add to your calendar*:
> > > >
> > > >
> > >
> >
> https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add
> > > >
> > > > I will post the meeting notes on the dev mailing list as well as
> > > Confluence
> > > > for archival purposes (example
> > > > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes
> >).
> > > >
> > > > Once we discuss various proposals next week, I recommend that for
> each
> > > > "workstream", we have an owner who would want to lead that
> workstream.
> > > For
> > > > items, that does not have an owner we can put those into Airflow 3
> Meta
> > > > issue <https://github.com/apache/airflow/issues/39593> or cross-link
> > > over
> > > > there so someone in the community can take it on. If we don't have an
> > > owner
> > > > who will commit to working on it, we park that item until we find the
> > > > owner.
> > > >
> > > > At the end of each call, I would solicit ideas for the agenda for the
> > > next
> > > > call and propose it to the broader group on the mailing list.
> > > >
> > > > Some of the items that should be discussed in the upcoming calls IMO:
> > > >
> > > >    - Agreeing on Principles
> > > >
> > > >    Based on the discussions, some potential items (all up for debate)
> > > >       - Considering Airflow 3.0 for early adopters and* breaking (and
> > > >       removing) things for AF 3.0*. Things can be re-added as needed
> in
> > > >       upcoming minor releases
> > > >       - Optimize to get *foundational pieces in* and not "let perfect
> > be
> > > >       the enemy of good"
> > > >       - Working on features that solidify Airflow as the* modern
> > > >       Orchestrator* that also has state of the art *support for Data,
> > AI
> > > &
> > > >       ML workloads*. This includes scalability & performance
> discussion
> > > >       - Set up the codebase for the next 5 years. This encompasses
> all
> > > the
> > > >       things we are discussing e.g removing MySQL to reduce the test
> > > > matrix,
> > > >       simplifying things architecturally, consolidating serialization
> > > > methods, etc
> > > >
> > > >       - Workstream & Stream Owners
> > > >    - Airflow 2 support policy including scope (feature vs bug fixes +
> > > >    security only) & support period
> > > >    - Separate discussions for each big workstream including one for
> > items
> > > >    to remove & refactor (e.g dropping MySQL)
> > > >    - Discussion to streamline the development of Airflow 3
> > > >       - Separating dev for Providers & Airflow (something Jarek
> already
> > > >       kick-started), and
> > > >       - Separate branch for Airflow 2
> > > >       - CI changes for the above
> > > >    - Finalize Scope + Timelines
> > > >    - Migration Utilities
> > > >    - Progress check-ins
> > > >
> > > > Looking forward to the exciting months ahead.
> > > >
> > > > Regards,
> > > > Kaxil
> > > >
> > > > On Mon, 13 May 2024 at 21:40, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > > >
> > > > > Declaring connections prior to task execution was already proposed
> in
> > > > AIP-1
> > > > > :-). At that time, I had in mind to communicate over IPC to the
> task
> > > the
> > > > > required settings. Registration could then happen with a manifest.
> > > Maybe
> > > > > during DAG serialization this could be obtained unobtrusively? The
> > > > benefit
> > > > > is that tasks become truly atomic or independent from Airflow as
> long
> > > as
> > > > > they communicate their exit codes (success, failed, and I think Ash
> > > had a
> > > > > couple of others in mind - the fewer the better).
> > > > >
> > > > > If you want two-way communication, maybe for variables as they can
> > > change
> > > > > during scheduling, this can happen with AIP-44. Although, I'd
> prefer
> > it
> > > > to
> > > > > happen with the *executor* rather than some centralized service. If
> > the
> > > > > executor is used, IPC is the logical choice. The benefit of this is
> > > that
> > > > > you have better resiliency and you can start to think about no
> > downtime
> > > > > upgrades
> > > > >
> > > > > So I hope Ash takes this to 2024 :-).
> > > > >
> > > > > B.
> > > > >
> > > > >
> > > > > On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <as...@apache.org>
> > > wrote:
> > > > >
> > > > > > > That would require some mechanism of declaring prior to task
> > > > execution
> > > > > > what connections would be used
> > > > > >
> > > > > > That’s exactly what I’m proposing in the proposal doc I’m working
> > on
> > > > > (It’s
> > > > > > part of also overhauling and re-designing the “Task Execution
> > > > interface”
> > > > > > that also gives us the ability to nicely have support for running
> > > tasks
> > > > > in
> > > > > > other languages — much more than just BashOperator)
> > > > > >
> > > > > > This is a bit of a fundamental shift in thinking about task
> > execution
> > > > in
> > > > > > Airflow, but I think it gives us some really nice properties that
> > the
> > > > > > project is currently missing.
> > > > > >
> > > > > > Tl;dr; lets discuss this in my doc when it comes our (next week
> > most
> > > > > > likely) please :)
> > > > > >
> > > > > > -ash
> > > > > >
> > > > > > > On 13 May 2024, at 18:15, Daniel Standish
> > > > > > <da...@astronomer.io.INVALID> wrote:
> > > > > > >
> > > > > > > re
> > > > > > >
> > > > > > > As tasks require connection access, I assume connection data
> will
> > > > > somehow
> > > > > > >> be passed as part of the
> > > > > > >> metadata to task execution - whether it's part of the executor
> > > > > protocol
> > > > > > or
> > > > > > >> in some other way (I'm
> > > > > > >> not an expert on that part of Airflow). Then, provided it's
> > > > accessible
> > > > > > as
> > > > > > >> part of some execution
> > > > > > >> context, and not only passed to the task's execute method,
> > > > OpenLineage
> > > > > > >> could utilize it.
> > > > > > >>
> > > > > > >
> > > > > > > It's not strictly necessary that connection info be passed "as
> > part
> > > > of
> > > > > > task
> > > > > > > matadata".  That would require some mechanism of declaring
> prior
> > to
> > > > > task
> > > > > > > execution what connections would be used.  This is a thought
> that
> > > has
> > > > > > come
> > > > > > > up when thinking about execution of non-python tasks.  But it's
> > not
> > > > > > > required from a technical perspective by AIP-44 because the
> > > > > > > `get_connection` function can be made to be an RPC call so a
> task
> > > > could
> > > > > > > continue to retrieve connections at runtime.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > --
> > > > > Bolke de Bruin
> > > > > bdbruin@gmail.com
> > > > >
> > > >
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Kaxil Naik <ka...@gmail.com>]

Yeah, that works for me

Can we have it possible to have two (or maybe three -
> like a sub-committee) co-owners of topics?


On Tue, 14 May 2024 at 06:15, Vikram Koka <vi...@astronomer.io.invalid>
wrote:

> Definitely a fast moving thread on the mailing list. I haven’t been able to
> respond for a few days and feel very far behind already.
>
> A few comments on topics discussed the last few days:
> - Jarek, in response to your comments around being more aggressive than in
> Airflow 2 about deprecation and drops of functionality, I am very
> supportive of that stance. I completely agree that we could have been more
> aggressive as part of Airflow 2.
> However, I would like to ask that as we go forward, we make sure that we
> have clean interfaces to be able to add support, even if we choose a single
> implementation. For example, with respect to dropping MySQL support. I can
> understand the perspective of the project that this should be deprecated
> from an Airflow OSS perspective. However, even if the only OSS supported DB
> is Postgres, I would like to ensure that a clean interface exists for
> interaction with the DB, so that other databases such as MySQL or others
> CAN be supported by a third party or at a later date.
> I realize that this may seem onerous, but I believe that it enables us to
> be more flexible in the long run, rather than locking us into a single DB
> implementation.
>
> - Bolke, Daniel Standish, Ash, et al on the task execution contract,
> definitely looking forward to this.
>
> - To those that I proposed a couple of more detailed write ups, I still
> plan to do that, at the latest by early next week.
>
> Vikram
>
>
>
> On Mon, May 13, 2024 at 9:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Super-excited about that.
> >
> > Question/Proposal: Can we have it possible to have two (or maybe three -
> > like a sub-committee) co-owners of topics? I think it's a lot to put on
> > one's head to "own" a topic and given circumstances/ volunteer time of
> > people, interruptions (and life intervening), it might be a bit risky to
> > put it on one's shoulders only.
> >
> > I know it's against the rule ("if it is owned by many, it's not owned by
> > anyone") - but I think in our case there are at least some topics that
> > could benefit from having more than one owner. Especially when we know
> and
> > trust that we can work together on some topics that we are passionate
> > about. It might also encourage getting out of people's comfort zones.
> >
> > For example - I'd absolutely love to volunteer to co-own the "streamline
> > the development" with Andrey if he would be willing to of course :D
> (sorry
> > Andrey for "volunteering you" on that one :D) - and maybe we could get
> > someone else to join us.
> >
> > That might have the added benefit of being able to break with the way
> > we've been doing things. If I am owning it for one - I'd likely gravitate
> > towards past choices, but with others joining me and taking decisions
> (and
> > responsibility in making sure we implement them) together, we could make
> > better decisions and reduce bus factor for dev tooling/ CI in the future.
> >
> > BTW.  Shameless promotion: tomorrow I am giving a talk about that very
> > topic (in the context of last few years not yet Airflow 3.0) at the NY
> > meetup hosted at Astronomer NY headquarters
> > https://www.meetup.com/nyc-apache-airflow-meetup/events/300017228/ - so
> if
> > you are in NY or around - I think you can stil sign up :D. I am also
> > getting to PyCon US in Pittsburgh next week so don't expect too much from
> > me. I will be gearing up for streamlining the development by talking to
> the
> > right people and listening to the latest things and best practices of the
> > larger Python community :).
> >
> > J.
> >
> > On Tue, May 14, 2024 at 12:03 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Thank you all, I am very happy about the discussions.
> > >
> > > The mailing list moves fast :). The main reason I recommended starting
> > the
> > > dev calls in early June was to have some of these discussions on the
> > > mailing list.
> > >
> > > Since Michal already scheduled a call, let's start there to discuss
> > > various ideas. For the week after that, I have created an Airflow
> 2-style
> > > recurring open dev calls for anyone to join, info below:
> > >
> > > *Date & Time: *Recurring every 2 weeks on Thursday at* 4pm BST *( 3 PM
> > > GMT/UTC | 11 AM EST | 8 AM PST); starting* May 30, 2024 04:00 PM BST*
> and
> > > then
> > > *One-time registration Link*:
> > >
> > >
> >
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> > > *Add to your calendar*:
> > >
> > >
> >
> https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add
> > >
> > > I will post the meeting notes on the dev mailing list as well as
> > Confluence
> > > for archival purposes (example
> > > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes>).
> > >
> > > Once we discuss various proposals next week, I recommend that for each
> > > "workstream", we have an owner who would want to lead that workstream.
> > For
> > > items, that does not have an owner we can put those into Airflow 3 Meta
> > > issue <https://github.com/apache/airflow/issues/39593> or cross-link
> > over
> > > there so someone in the community can take it on. If we don't have an
> > owner
> > > who will commit to working on it, we park that item until we find the
> > > owner.
> > >
> > > At the end of each call, I would solicit ideas for the agenda for the
> > next
> > > call and propose it to the broader group on the mailing list.
> > >
> > > Some of the items that should be discussed in the upcoming calls IMO:
> > >
> > >    - Agreeing on Principles
> > >
> > >    Based on the discussions, some potential items (all up for debate)
> > >       - Considering Airflow 3.0 for early adopters and* breaking (and
> > >       removing) things for AF 3.0*. Things can be re-added as needed in
> > >       upcoming minor releases
> > >       - Optimize to get *foundational pieces in* and not "let perfect
> be
> > >       the enemy of good"
> > >       - Working on features that solidify Airflow as the* modern
> > >       Orchestrator* that also has state of the art *support for Data,
> AI
> > &
> > >       ML workloads*. This includes scalability & performance discussion
> > >       - Set up the codebase for the next 5 years. This encompasses all
> > the
> > >       things we are discussing e.g removing MySQL to reduce the test
> > > matrix,
> > >       simplifying things architecturally, consolidating serialization
> > > methods, etc
> > >
> > >       - Workstream & Stream Owners
> > >    - Airflow 2 support policy including scope (feature vs bug fixes +
> > >    security only) & support period
> > >    - Separate discussions for each big workstream including one for
> items
> > >    to remove & refactor (e.g dropping MySQL)
> > >    - Discussion to streamline the development of Airflow 3
> > >       - Separating dev for Providers & Airflow (something Jarek already
> > >       kick-started), and
> > >       - Separate branch for Airflow 2
> > >       - CI changes for the above
> > >    - Finalize Scope + Timelines
> > >    - Migration Utilities
> > >    - Progress check-ins
> > >
> > > Looking forward to the exciting months ahead.
> > >
> > > Regards,
> > > Kaxil
> > >
> > > On Mon, 13 May 2024 at 21:40, Bolke de Bruin <bd...@gmail.com>
> wrote:
> > >
> > > > Declaring connections prior to task execution was already proposed in
> > > AIP-1
> > > > :-). At that time, I had in mind to communicate over IPC to the task
> > the
> > > > required settings. Registration could then happen with a manifest.
> > Maybe
> > > > during DAG serialization this could be obtained unobtrusively? The
> > > benefit
> > > > is that tasks become truly atomic or independent from Airflow as long
> > as
> > > > they communicate their exit codes (success, failed, and I think Ash
> > had a
> > > > couple of others in mind - the fewer the better).
> > > >
> > > > If you want two-way communication, maybe for variables as they can
> > change
> > > > during scheduling, this can happen with AIP-44. Although, I'd prefer
> it
> > > to
> > > > happen with the *executor* rather than some centralized service. If
> the
> > > > executor is used, IPC is the logical choice. The benefit of this is
> > that
> > > > you have better resiliency and you can start to think about no
> downtime
> > > > upgrades
> > > >
> > > > So I hope Ash takes this to 2024 :-).
> > > >
> > > > B.
> > > >
> > > >
> > > > On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > > >
> > > > > > That would require some mechanism of declaring prior to task
> > > execution
> > > > > what connections would be used
> > > > >
> > > > > That’s exactly what I’m proposing in the proposal doc I’m working
> on
> > > > (It’s
> > > > > part of also overhauling and re-designing the “Task Execution
> > > interface”
> > > > > that also gives us the ability to nicely have support for running
> > tasks
> > > > in
> > > > > other languages — much more than just BashOperator)
> > > > >
> > > > > This is a bit of a fundamental shift in thinking about task
> execution
> > > in
> > > > > Airflow, but I think it gives us some really nice properties that
> the
> > > > > project is currently missing.
> > > > >
> > > > > Tl;dr; lets discuss this in my doc when it comes our (next week
> most
> > > > > likely) please :)
> > > > >
> > > > > -ash
> > > > >
> > > > > > On 13 May 2024, at 18:15, Daniel Standish
> > > > > <da...@astronomer.io.INVALID> wrote:
> > > > > >
> > > > > > re
> > > > > >
> > > > > > As tasks require connection access, I assume connection data will
> > > > somehow
> > > > > >> be passed as part of the
> > > > > >> metadata to task execution - whether it's part of the executor
> > > > protocol
> > > > > or
> > > > > >> in some other way (I'm
> > > > > >> not an expert on that part of Airflow). Then, provided it's
> > > accessible
> > > > > as
> > > > > >> part of some execution
> > > > > >> context, and not only passed to the task's execute method,
> > > OpenLineage
> > > > > >> could utilize it.
> > > > > >>
> > > > > >
> > > > > > It's not strictly necessary that connection info be passed "as
> part
> > > of
> > > > > task
> > > > > > matadata".  That would require some mechanism of declaring prior
> to
> > > > task
> > > > > > execution what connections would be used.  This is a thought that
> > has
> > > > > come
> > > > > > up when thinking about execution of non-python tasks.  But it's
> not
> > > > > > required from a technical perspective by AIP-44 because the
> > > > > > `get_connection` function can be made to be an RPC call so a task
> > > could
> > > > > > continue to retrieve connections at runtime.
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > --
> > > > Bolke de Bruin
> > > > bdbruin@gmail.com
> > > >
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Vikram Koka <vi...@astronomer.io.INVALID>]

Definitely a fast moving thread on the mailing list. I haven’t been able to
respond for a few days and feel very far behind already.

A few comments on topics discussed the last few days:
- Jarek, in response to your comments around being more aggressive than in
Airflow 2 about deprecation and drops of functionality, I am very
supportive of that stance. I completely agree that we could have been more
aggressive as part of Airflow 2.
However, I would like to ask that as we go forward, we make sure that we
have clean interfaces to be able to add support, even if we choose a single
implementation. For example, with respect to dropping MySQL support. I can
understand the perspective of the project that this should be deprecated
from an Airflow OSS perspective. However, even if the only OSS supported DB
is Postgres, I would like to ensure that a clean interface exists for
interaction with the DB, so that other databases such as MySQL or others
CAN be supported by a third party or at a later date.
I realize that this may seem onerous, but I believe that it enables us to
be more flexible in the long run, rather than locking us into a single DB
implementation.

- Bolke, Daniel Standish, Ash, et al on the task execution contract,
definitely looking forward to this.

- To those that I proposed a couple of more detailed write ups, I still
plan to do that, at the latest by early next week.

Vikram



On Mon, May 13, 2024 at 9:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Super-excited about that.
>
> Question/Proposal: Can we have it possible to have two (or maybe three -
> like a sub-committee) co-owners of topics? I think it's a lot to put on
> one's head to "own" a topic and given circumstances/ volunteer time of
> people, interruptions (and life intervening), it might be a bit risky to
> put it on one's shoulders only.
>
> I know it's against the rule ("if it is owned by many, it's not owned by
> anyone") - but I think in our case there are at least some topics that
> could benefit from having more than one owner. Especially when we know and
> trust that we can work together on some topics that we are passionate
> about. It might also encourage getting out of people's comfort zones.
>
> For example - I'd absolutely love to volunteer to co-own the "streamline
> the development" with Andrey if he would be willing to of course :D (sorry
> Andrey for "volunteering you" on that one :D) - and maybe we could get
> someone else to join us.
>
> That might have the added benefit of being able to break with the way
> we've been doing things. If I am owning it for one - I'd likely gravitate
> towards past choices, but with others joining me and taking decisions (and
> responsibility in making sure we implement them) together, we could make
> better decisions and reduce bus factor for dev tooling/ CI in the future.
>
> BTW.  Shameless promotion: tomorrow I am giving a talk about that very
> topic (in the context of last few years not yet Airflow 3.0) at the NY
> meetup hosted at Astronomer NY headquarters
> https://www.meetup.com/nyc-apache-airflow-meetup/events/300017228/ - so if
> you are in NY or around - I think you can stil sign up :D. I am also
> getting to PyCon US in Pittsburgh next week so don't expect too much from
> me. I will be gearing up for streamlining the development by talking to the
> right people and listening to the latest things and best practices of the
> larger Python community :).
>
> J.
>
> On Tue, May 14, 2024 at 12:03 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Thank you all, I am very happy about the discussions.
> >
> > The mailing list moves fast :). The main reason I recommended starting
> the
> > dev calls in early June was to have some of these discussions on the
> > mailing list.
> >
> > Since Michal already scheduled a call, let's start there to discuss
> > various ideas. For the week after that, I have created an Airflow 2-style
> > recurring open dev calls for anyone to join, info below:
> >
> > *Date & Time: *Recurring every 2 weeks on Thursday at* 4pm BST *( 3 PM
> > GMT/UTC | 11 AM EST | 8 AM PST); starting* May 30, 2024 04:00 PM BST* and
> > then
> > *One-time registration Link*:
> >
> >
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> > *Add to your calendar*:
> >
> >
> https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add
> >
> > I will post the meeting notes on the dev mailing list as well as
> Confluence
> > for archival purposes (example
> > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes>).
> >
> > Once we discuss various proposals next week, I recommend that for each
> > "workstream", we have an owner who would want to lead that workstream.
> For
> > items, that does not have an owner we can put those into Airflow 3 Meta
> > issue <https://github.com/apache/airflow/issues/39593> or cross-link
> over
> > there so someone in the community can take it on. If we don't have an
> owner
> > who will commit to working on it, we park that item until we find the
> > owner.
> >
> > At the end of each call, I would solicit ideas for the agenda for the
> next
> > call and propose it to the broader group on the mailing list.
> >
> > Some of the items that should be discussed in the upcoming calls IMO:
> >
> >    - Agreeing on Principles
> >
> >    Based on the discussions, some potential items (all up for debate)
> >       - Considering Airflow 3.0 for early adopters and* breaking (and
> >       removing) things for AF 3.0*. Things can be re-added as needed in
> >       upcoming minor releases
> >       - Optimize to get *foundational pieces in* and not "let perfect be
> >       the enemy of good"
> >       - Working on features that solidify Airflow as the* modern
> >       Orchestrator* that also has state of the art *support for Data, AI
> &
> >       ML workloads*. This includes scalability & performance discussion
> >       - Set up the codebase for the next 5 years. This encompasses all
> the
> >       things we are discussing e.g removing MySQL to reduce the test
> > matrix,
> >       simplifying things architecturally, consolidating serialization
> > methods, etc
> >
> >       - Workstream & Stream Owners
> >    - Airflow 2 support policy including scope (feature vs bug fixes +
> >    security only) & support period
> >    - Separate discussions for each big workstream including one for items
> >    to remove & refactor (e.g dropping MySQL)
> >    - Discussion to streamline the development of Airflow 3
> >       - Separating dev for Providers & Airflow (something Jarek already
> >       kick-started), and
> >       - Separate branch for Airflow 2
> >       - CI changes for the above
> >    - Finalize Scope + Timelines
> >    - Migration Utilities
> >    - Progress check-ins
> >
> > Looking forward to the exciting months ahead.
> >
> > Regards,
> > Kaxil
> >
> > On Mon, 13 May 2024 at 21:40, Bolke de Bruin <bd...@gmail.com> wrote:
> >
> > > Declaring connections prior to task execution was already proposed in
> > AIP-1
> > > :-). At that time, I had in mind to communicate over IPC to the task
> the
> > > required settings. Registration could then happen with a manifest.
> Maybe
> > > during DAG serialization this could be obtained unobtrusively? The
> > benefit
> > > is that tasks become truly atomic or independent from Airflow as long
> as
> > > they communicate their exit codes (success, failed, and I think Ash
> had a
> > > couple of others in mind - the fewer the better).
> > >
> > > If you want two-way communication, maybe for variables as they can
> change
> > > during scheduling, this can happen with AIP-44. Although, I'd prefer it
> > to
> > > happen with the *executor* rather than some centralized service. If the
> > > executor is used, IPC is the logical choice. The benefit of this is
> that
> > > you have better resiliency and you can start to think about no downtime
> > > upgrades
> > >
> > > So I hope Ash takes this to 2024 :-).
> > >
> > > B.
> > >
> > >
> > > On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <as...@apache.org>
> wrote:
> > >
> > > > > That would require some mechanism of declaring prior to task
> > execution
> > > > what connections would be used
> > > >
> > > > That’s exactly what I’m proposing in the proposal doc I’m working on
> > > (It’s
> > > > part of also overhauling and re-designing the “Task Execution
> > interface”
> > > > that also gives us the ability to nicely have support for running
> tasks
> > > in
> > > > other languages — much more than just BashOperator)
> > > >
> > > > This is a bit of a fundamental shift in thinking about task execution
> > in
> > > > Airflow, but I think it gives us some really nice properties that the
> > > > project is currently missing.
> > > >
> > > > Tl;dr; lets discuss this in my doc when it comes our (next week most
> > > > likely) please :)
> > > >
> > > > -ash
> > > >
> > > > > On 13 May 2024, at 18:15, Daniel Standish
> > > > <da...@astronomer.io.INVALID> wrote:
> > > > >
> > > > > re
> > > > >
> > > > > As tasks require connection access, I assume connection data will
> > > somehow
> > > > >> be passed as part of the
> > > > >> metadata to task execution - whether it's part of the executor
> > > protocol
> > > > or
> > > > >> in some other way (I'm
> > > > >> not an expert on that part of Airflow). Then, provided it's
> > accessible
> > > > as
> > > > >> part of some execution
> > > > >> context, and not only passed to the task's execute method,
> > OpenLineage
> > > > >> could utilize it.
> > > > >>
> > > > >
> > > > > It's not strictly necessary that connection info be passed "as part
> > of
> > > > task
> > > > > matadata".  That would require some mechanism of declaring prior to
> > > task
> > > > > execution what connections would be used.  This is a thought that
> has
> > > > come
> > > > > up when thinking about execution of non-python tasks.  But it's not
> > > > > required from a technical perspective by AIP-44 because the
> > > > > `get_connection` function can be made to be an RPC call so a task
> > could
> > > > > continue to retrieve connections at runtime.
> > > >
> > > >
> > >
> > > --
> > >
> > > --
> > > Bolke de Bruin
> > > bdbruin@gmail.com
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Jarek Potiuk <ja...@potiuk.com>]

Super-excited about that.

Question/Proposal: Can we have it possible to have two (or maybe three -
like a sub-committee) co-owners of topics? I think it's a lot to put on
one's head to "own" a topic and given circumstances/ volunteer time of
people, interruptions (and life intervening), it might be a bit risky to
put it on one's shoulders only.

I know it's against the rule ("if it is owned by many, it's not owned by
anyone") - but I think in our case there are at least some topics that
could benefit from having more than one owner. Especially when we know and
trust that we can work together on some topics that we are passionate
about. It might also encourage getting out of people's comfort zones.

For example - I'd absolutely love to volunteer to co-own the "streamline
the development" with Andrey if he would be willing to of course :D (sorry
Andrey for "volunteering you" on that one :D) - and maybe we could get
someone else to join us.

That might have the added benefit of being able to break with the way
we've been doing things. If I am owning it for one - I'd likely gravitate
towards past choices, but with others joining me and taking decisions (and
responsibility in making sure we implement them) together, we could make
better decisions and reduce bus factor for dev tooling/ CI in the future.

BTW.  Shameless promotion: tomorrow I am giving a talk about that very
topic (in the context of last few years not yet Airflow 3.0) at the NY
meetup hosted at Astronomer NY headquarters
https://www.meetup.com/nyc-apache-airflow-meetup/events/300017228/ - so if
you are in NY or around - I think you can stil sign up :D. I am also
getting to PyCon US in Pittsburgh next week so don't expect too much from
me. I will be gearing up for streamlining the development by talking to the
right people and listening to the latest things and best practices of the
larger Python community :).

J.

On Tue, May 14, 2024 at 12:03 AM Kaxil Naik <ka...@gmail.com> wrote:

> Thank you all, I am very happy about the discussions.
>
> The mailing list moves fast :). The main reason I recommended starting the
> dev calls in early June was to have some of these discussions on the
> mailing list.
>
> Since Michal already scheduled a call, let's start there to discuss
> various ideas. For the week after that, I have created an Airflow 2-style
> recurring open dev calls for anyone to join, info below:
>
> *Date & Time: *Recurring every 2 weeks on Thursday at* 4pm BST *( 3 PM
> GMT/UTC | 11 AM EST | 8 AM PST); starting* May 30, 2024 04:00 PM BST* and
> then
> *One-time registration Link*:
>
> https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
> *Add to your calendar*:
>
> https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add
>
> I will post the meeting notes on the dev mailing list as well as Confluence
> for archival purposes (example
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes>).
>
> Once we discuss various proposals next week, I recommend that for each
> "workstream", we have an owner who would want to lead that workstream. For
> items, that does not have an owner we can put those into Airflow 3 Meta
> issue <https://github.com/apache/airflow/issues/39593> or cross-link over
> there so someone in the community can take it on. If we don't have an owner
> who will commit to working on it, we park that item until we find the
> owner.
>
> At the end of each call, I would solicit ideas for the agenda for the next
> call and propose it to the broader group on the mailing list.
>
> Some of the items that should be discussed in the upcoming calls IMO:
>
>    - Agreeing on Principles
>
>    Based on the discussions, some potential items (all up for debate)
>       - Considering Airflow 3.0 for early adopters and* breaking (and
>       removing) things for AF 3.0*. Things can be re-added as needed in
>       upcoming minor releases
>       - Optimize to get *foundational pieces in* and not "let perfect be
>       the enemy of good"
>       - Working on features that solidify Airflow as the* modern
>       Orchestrator* that also has state of the art *support for Data, AI &
>       ML workloads*. This includes scalability & performance discussion
>       - Set up the codebase for the next 5 years. This encompasses all the
>       things we are discussing e.g removing MySQL to reduce the test
> matrix,
>       simplifying things architecturally, consolidating serialization
> methods, etc
>
>       - Workstream & Stream Owners
>    - Airflow 2 support policy including scope (feature vs bug fixes +
>    security only) & support period
>    - Separate discussions for each big workstream including one for items
>    to remove & refactor (e.g dropping MySQL)
>    - Discussion to streamline the development of Airflow 3
>       - Separating dev for Providers & Airflow (something Jarek already
>       kick-started), and
>       - Separate branch for Airflow 2
>       - CI changes for the above
>    - Finalize Scope + Timelines
>    - Migration Utilities
>    - Progress check-ins
>
> Looking forward to the exciting months ahead.
>
> Regards,
> Kaxil
>
> On Mon, 13 May 2024 at 21:40, Bolke de Bruin <bd...@gmail.com> wrote:
>
> > Declaring connections prior to task execution was already proposed in
> AIP-1
> > :-). At that time, I had in mind to communicate over IPC to the task the
> > required settings. Registration could then happen with a manifest. Maybe
> > during DAG serialization this could be obtained unobtrusively? The
> benefit
> > is that tasks become truly atomic or independent from Airflow as long as
> > they communicate their exit codes (success, failed, and I think Ash had a
> > couple of others in mind - the fewer the better).
> >
> > If you want two-way communication, maybe for variables as they can change
> > during scheduling, this can happen with AIP-44. Although, I'd prefer it
> to
> > happen with the *executor* rather than some centralized service. If the
> > executor is used, IPC is the logical choice. The benefit of this is that
> > you have better resiliency and you can start to think about no downtime
> > upgrades
> >
> > So I hope Ash takes this to 2024 :-).
> >
> > B.
> >
> >
> > On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > > > That would require some mechanism of declaring prior to task
> execution
> > > what connections would be used
> > >
> > > That’s exactly what I’m proposing in the proposal doc I’m working on
> > (It’s
> > > part of also overhauling and re-designing the “Task Execution
> interface”
> > > that also gives us the ability to nicely have support for running tasks
> > in
> > > other languages — much more than just BashOperator)
> > >
> > > This is a bit of a fundamental shift in thinking about task execution
> in
> > > Airflow, but I think it gives us some really nice properties that the
> > > project is currently missing.
> > >
> > > Tl;dr; lets discuss this in my doc when it comes our (next week most
> > > likely) please :)
> > >
> > > -ash
> > >
> > > > On 13 May 2024, at 18:15, Daniel Standish
> > > <da...@astronomer.io.INVALID> wrote:
> > > >
> > > > re
> > > >
> > > > As tasks require connection access, I assume connection data will
> > somehow
> > > >> be passed as part of the
> > > >> metadata to task execution - whether it's part of the executor
> > protocol
> > > or
> > > >> in some other way (I'm
> > > >> not an expert on that part of Airflow). Then, provided it's
> accessible
> > > as
> > > >> part of some execution
> > > >> context, and not only passed to the task's execute method,
> OpenLineage
> > > >> could utilize it.
> > > >>
> > > >
> > > > It's not strictly necessary that connection info be passed "as part
> of
> > > task
> > > > matadata".  That would require some mechanism of declaring prior to
> > task
> > > > execution what connections would be used.  This is a thought that has
> > > come
> > > > up when thinking about execution of non-python tasks.  But it's not
> > > > required from a technical perspective by AIP-44 because the
> > > > `get_connection` function can be made to be an RPC call so a task
> could
> > > > continue to retrieve connections at runtime.
> > >
> > >
> >
> > --
> >
> > --
> > Bolke de Bruin
> > bdbruin@gmail.com
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Kaxil Naik <ka...@gmail.com>]

Thank you all, I am very happy about the discussions.

The mailing list moves fast :). The main reason I recommended starting the
dev calls in early June was to have some of these discussions on the
mailing list.

Since Michal already scheduled a call, let's start there to discuss
various ideas. For the week after that, I have created an Airflow 2-style
recurring open dev calls for anyone to join, info below:

*Date & Time: *Recurring every 2 weeks on Thursday at* 4pm BST *( 3 PM
GMT/UTC | 11 AM EST | 8 AM PST); starting* May 30, 2024 04:00 PM BST* and
then
*One-time registration Link*:
https://astronomer.zoom.us/meeting/register/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG
*Add to your calendar*:
https://astronomer.zoom.us/meeting/tZAsde2vqDwpE9XrBAbCeIFHA_l7OLywrWkG/calendar/google/add

I will post the meeting notes on the dev mailing list as well as Confluence
for archival purposes (example
<https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes>).

Once we discuss various proposals next week, I recommend that for each
"workstream", we have an owner who would want to lead that workstream. For
items, that does not have an owner we can put those into Airflow 3 Meta
issue <https://github.com/apache/airflow/issues/39593> or cross-link over
there so someone in the community can take it on. If we don't have an owner
who will commit to working on it, we park that item until we find the owner.

At the end of each call, I would solicit ideas for the agenda for the next
call and propose it to the broader group on the mailing list.

Some of the items that should be discussed in the upcoming calls IMO:

   - Agreeing on Principles

   Based on the discussions, some potential items (all up for debate)
      - Considering Airflow 3.0 for early adopters and* breaking (and
      removing) things for AF 3.0*. Things can be re-added as needed in
      upcoming minor releases
      - Optimize to get *foundational pieces in* and not "let perfect be
      the enemy of good"
      - Working on features that solidify Airflow as the* modern
      Orchestrator* that also has state of the art *support for Data, AI &
      ML workloads*. This includes scalability & performance discussion
      - Set up the codebase for the next 5 years. This encompasses all the
      things we are discussing e.g removing MySQL to reduce the test matrix,
      simplifying things architecturally, consolidating serialization
methods, etc

      - Workstream & Stream Owners
   - Airflow 2 support policy including scope (feature vs bug fixes +
   security only) & support period
   - Separate discussions for each big workstream including one for items
   to remove & refactor (e.g dropping MySQL)
   - Discussion to streamline the development of Airflow 3
      - Separating dev for Providers & Airflow (something Jarek already
      kick-started), and
      - Separate branch for Airflow 2
      - CI changes for the above
   - Finalize Scope + Timelines
   - Migration Utilities
   - Progress check-ins

Looking forward to the exciting months ahead.

Regards,
Kaxil

On Mon, 13 May 2024 at 21:40, Bolke de Bruin <bd...@gmail.com> wrote:

> Declaring connections prior to task execution was already proposed in AIP-1
> :-). At that time, I had in mind to communicate over IPC to the task the
> required settings. Registration could then happen with a manifest. Maybe
> during DAG serialization this could be obtained unobtrusively? The benefit
> is that tasks become truly atomic or independent from Airflow as long as
> they communicate their exit codes (success, failed, and I think Ash had a
> couple of others in mind - the fewer the better).
>
> If you want two-way communication, maybe for variables as they can change
> during scheduling, this can happen with AIP-44. Although, I'd prefer it to
> happen with the *executor* rather than some centralized service. If the
> executor is used, IPC is the logical choice. The benefit of this is that
> you have better resiliency and you can start to think about no downtime
> upgrades
>
> So I hope Ash takes this to 2024 :-).
>
> B.
>
>
> On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > > That would require some mechanism of declaring prior to task execution
> > what connections would be used
> >
> > That’s exactly what I’m proposing in the proposal doc I’m working on
> (It’s
> > part of also overhauling and re-designing the “Task Execution interface”
> > that also gives us the ability to nicely have support for running tasks
> in
> > other languages — much more than just BashOperator)
> >
> > This is a bit of a fundamental shift in thinking about task execution in
> > Airflow, but I think it gives us some really nice properties that the
> > project is currently missing.
> >
> > Tl;dr; lets discuss this in my doc when it comes our (next week most
> > likely) please :)
> >
> > -ash
> >
> > > On 13 May 2024, at 18:15, Daniel Standish
> > <da...@astronomer.io.INVALID> wrote:
> > >
> > > re
> > >
> > > As tasks require connection access, I assume connection data will
> somehow
> > >> be passed as part of the
> > >> metadata to task execution - whether it's part of the executor
> protocol
> > or
> > >> in some other way (I'm
> > >> not an expert on that part of Airflow). Then, provided it's accessible
> > as
> > >> part of some execution
> > >> context, and not only passed to the task's execute method, OpenLineage
> > >> could utilize it.
> > >>
> > >
> > > It's not strictly necessary that connection info be passed "as part of
> > task
> > > matadata".  That would require some mechanism of declaring prior to
> task
> > > execution what connections would be used.  This is a thought that has
> > come
> > > up when thinking about execution of non-python tasks.  But it's not
> > > required from a technical perspective by AIP-44 because the
> > > `get_connection` function can be made to be an RPC call so a task could
> > > continue to retrieve connections at runtime.
> >
> >
>
> --
>
> --
> Bolke de Bruin
> bdbruin@gmail.com
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Bolke de Bruin <bd...@gmail.com>]

Declaring connections prior to task execution was already proposed in AIP-1
:-). At that time, I had in mind to communicate over IPC to the task the
required settings. Registration could then happen with a manifest. Maybe
during DAG serialization this could be obtained unobtrusively? The benefit
is that tasks become truly atomic or independent from Airflow as long as
they communicate their exit codes (success, failed, and I think Ash had a
couple of others in mind - the fewer the better).

If you want two-way communication, maybe for variables as they can change
during scheduling, this can happen with AIP-44. Although, I'd prefer it to
happen with the *executor* rather than some centralized service. If the
executor is used, IPC is the logical choice. The benefit of this is that
you have better resiliency and you can start to think about no downtime
upgrades

So I hope Ash takes this to 2024 :-).

B.


On Mon, 13 May 2024 at 19:27, Ash Berlin-Taylor <as...@apache.org> wrote:

> > That would require some mechanism of declaring prior to task execution
> what connections would be used
>
> That’s exactly what I’m proposing in the proposal doc I’m working on (It’s
> part of also overhauling and re-designing the “Task Execution interface”
> that also gives us the ability to nicely have support for running tasks in
> other languages — much more than just BashOperator)
>
> This is a bit of a fundamental shift in thinking about task execution in
> Airflow, but I think it gives us some really nice properties that the
> project is currently missing.
>
> Tl;dr; lets discuss this in my doc when it comes our (next week most
> likely) please :)
>
> -ash
>
> > On 13 May 2024, at 18:15, Daniel Standish
> <da...@astronomer.io.INVALID> wrote:
> >
> > re
> >
> > As tasks require connection access, I assume connection data will somehow
> >> be passed as part of the
> >> metadata to task execution - whether it's part of the executor protocol
> or
> >> in some other way (I'm
> >> not an expert on that part of Airflow). Then, provided it's accessible
> as
> >> part of some execution
> >> context, and not only passed to the task's execute method, OpenLineage
> >> could utilize it.
> >>
> >
> > It's not strictly necessary that connection info be passed "as part of
> task
> > matadata".  That would require some mechanism of declaring prior to task
> > execution what connections would be used.  This is a thought that has
> come
> > up when thinking about execution of non-python tasks.  But it's not
> > required from a technical perspective by AIP-44 because the
> > `get_connection` function can be made to be an RPC call so a task could
> > continue to retrieve connections at runtime.
>
>

-- 

--
Bolke de Bruin
bdbruin@gmail.com

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Ash Berlin-Taylor <as...@apache.org>]

> That would require some mechanism of declaring prior to task execution what connections would be used

That’s exactly what I’m proposing in the proposal doc I’m working on (It’s part of also overhauling and re-designing the “Task Execution interface” that also gives us the ability to nicely have support for running tasks in other languages — much more than just BashOperator)

This is a bit of a fundamental shift in thinking about task execution in Airflow, but I think it gives us some really nice properties that the project is currently missing.

Tl;dr; lets discuss this in my doc when it comes our (next week most likely) please :)

-ash

> On 13 May 2024, at 18:15, Daniel Standish <da...@astronomer.io.INVALID> wrote:
> 
> re
> 
> As tasks require connection access, I assume connection data will somehow
>> be passed as part of the
>> metadata to task execution - whether it's part of the executor protocol or
>> in some other way (I'm
>> not an expert on that part of Airflow). Then, provided it's accessible as
>> part of some execution
>> context, and not only passed to the task's execute method, OpenLineage
>> could utilize it.
>> 
> 
> It's not strictly necessary that connection info be passed "as part of task
> matadata".  That would require some mechanism of declaring prior to task
> execution what connections would be used.  This is a thought that has come
> up when thinking about execution of non-python tasks.  But it's not
> required from a technical perspective by AIP-44 because the
> `get_connection` function can be made to be an RPC call so a task could
> continue to retrieve connections at runtime.

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Daniel Standish <da...@astronomer.io.INVALID>]

re

As tasks require connection access, I assume connection data will somehow
> be passed as part of the
> metadata to task execution - whether it's part of the executor protocol or
> in some other way (I'm
> not an expert on that part of Airflow). Then, provided it's accessible as
> part of some execution
> context, and not only passed to the task's execute method, OpenLineage
> could utilize it.
>

It's not strictly necessary that connection info be passed "as part of task
matadata".  That would require some mechanism of declaring prior to task
execution what connections would be used.  This is a thought that has come
up when thinking about execution of non-python tasks.  But it's not
required from a technical perspective by AIP-44 because the
`get_connection` function can be made to be an RPC call so a task could
continue to retrieve connections at runtime.

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Michał Modras <mi...@google.com.INVALID>]

I think relying on connections wrapped in the task execution context makes
sense as a way to retrieve the connections to be used by OpenLineage.
Similarly, lineage backend relied on the task instance context in the past (
https://github.com/apache/airflow/blob/dc61da2509536bb509a4a2e1781ff47aa36b62b4/airflow/lineage/backend.py#L44;
not sure if connections were included in it though). It also has the
benefit of the appropriate connections (and authorized for the task) being
present in the task context, and not querying db for connections, having
access to all that are defined. One thing we'd need to verify is how that
context is serialised/persisted (if it is at all), so that we don't
proliferate any connection data.

On Mon, May 13, 2024 at 10:53 AM Maciej Obuchowski <
obuchowski.maciej@gmail.com> wrote:

> I would like to discuss the role of the Listener API and the OpenLineage
> provider that utilizes it,
> as there are current discussions that impact this integration.
>
> For example, there is a consensus on (and I strongly support) removing
> direct database access from
> workers. However, currently, OpenLineage integration depends on it. I want
> to explain what we do
> and how we can proceed with removing database access from workers while not
> only retaining but
> possibly enhancing lineage functionality.
>
> Currently, OpenLineage uses not only the data that the task directly has
> but also utilizes
> external connections in three ways:
>
> - We send events to the configured OpenLineage backend. This does not
> depend on any external
> connection or Airflow database but merely on configuration, so it's
> acceptable.
> - We use the Airflow database, primarily to get connections - for example,
> when gathering lineage
> from SQLExecuteQueryOperator, to obtain information about the default
> database or schema.
> - We use external APIs - sometimes directly (like asking BigQuery about job
> table dependencies) or
> using Airflow Connections received from the Airflow database, like in
> SQLExecuteQueryOperator to
> get table schemas.
>
> The issue arises if we are unable to perform the second or third methods -
> not being able to
> understand on which database a SELECT * from table operates would render
> the potential lineage
> data gathered from the statement itself meaningless.
>
> As tasks require connection access, I assume connection data will somehow
> be passed as part of the
> metadata to task execution - whether it's part of the executor protocol or
> in some other way (I'm
> not an expert on that part of Airflow). Then, provided it's accessible as
> part of some execution
> context, and not only passed to the task's execute method, OpenLineage
> could utilize it.
>
> If the task execution metadata were standardized, we could take a step
> further and perhaps move
> from a model where TaskInstance listeners are executed on the worker to one
> where they simply
> depend on that metadata rather than the TaskInstance object. They could be
> executed elsewhere - on
> some specialized component working asynchronously, like the triggerer -
> which would not be a fitting
> name anymore.
>
> pon., 13 maj 2024 o 08:55 Michał Modras <mi...@google.com.invalid>
> napisał(a):
>
> > Thanks to everyone for the constructive discussion - let's make it even
> > more specific in the weekly syncs! I put a slot for us next Tuesday. I
> had
> > some challenges with how some of the e-mails are displayed in this
> thread -
> > please reach out to me if you'd like to be added.
> >
> > In the meantime, I summarised challenges, opportunities and potential
> > work streams of Airflow 3 from the Composer team perspective. Please
> have a
> > look and chime in here:
> >
> >
> https://docs.google.com/document/d/1kfqq3YFtUjFuuiv7KVZRqYgwFu9jcYyW_Mz7UUZYnks/edit?usp=sharing
> >
> > The main themes are on-demand & event driven DAG parsing, performance,
> > isolation and stability improvements, and a big concern of backwards
> > compatibility (+1 to Shubham's point - let's try to make changes in DAGS
> > unnecessary, as opposed to potential changes in Airflow configuration).
> >
> > Best,
> > Michal
> >
> > On Sat, May 11, 2024 at 9:30 AM Mehta, Shubham <shubhx@amazon.com.invalid
> >
> > wrote:
> >
> > > Sorry for joining the discussion late - I was down under the weather
> for
> > > the past two weeks.
> > > First off, great discussion so far and appreciate the effort and
> thought
> > > everyone has put into exploring Airflow 3.
> > >
> > > The list from Kaxil, et al looks solid and I'm really looking forward
> to
> > > more extensive discussions on those and seeing them implemented in
> > Airflow.
> > > They will truly unlock some amazing use cases with Airflow.
> > >
> > > Regarding building for Airflow 2 vs Airflow 3, personally, I think we
> > > should keep pushing critical features for Airflow 2 until the scope for
> > > Airflow 3 is properly defined. For example, we shouldn't pause work on
> > the
> > > multi-team capabilities as there are quite a lot of enterprises rely on
> > > Airflow today, and we have been saying that this feature is coming for
> > > years now. Introducing this in Airflow 3 means users have to jump
> through
> > > another non-backward compatible release to take advantage of it, which
> > > doesn't feel user focused. Airflow 3 is still at least a year away from
> > > anything substantial shipping. Not every change has to be breaking - we
> > can
> > > develop some things for Airflow 2 and potentially adopt them as-is in
> > > Airflow 3 if it makes sense. Only if we're certain the dev effort will
> be
> > > way less by starting fresh on Airflow 3, should we consider that path.
> > >
> > > On Jarek's point about being very opinionated in Airflow 3.0 and
> > > potentially dropping certain features/options to start with a lean core
> > - I
> > > am very divided on this. In most cases, I would say that being
> > opinionated
> > > is great, but in this case, we risk alienating our users, especially
> > those
> > > who self-host as they have proportionately less representation among
> the
> > > maintainers. Thousands of organizations rely on Airflow and breaking
> the
> > > trust could be costly in the long run. We should of course be basing
> > those
> > > decisions on real user data versus assumptions, but not everyone is
> going
> > > to jump to 2.10+ in a few months and data will be biased. I am not
> > > concerned about MWAA customers here, but more about users who are
> > > self-hosting.
> > >
> > > We've put together a Google doc (with writing access for all) capturing
> > > some wish list items based on customer discussions:
> > >
> >
> https://docs.google.com/document/d/17KKjwmwHquL_laMpsqZaZD5wzruKAAyFeOsLhGNp7vc/edit?usp=sharing
> > > . It's still a work-in-progress as we gather more feedback from our
> > > customers on what could help increase Airflow adoption in their orgs.
> We
> > > understand that not everything in this list will be implemented, but it
> > is
> > > essential to bring forward all ideas and discuss to make Airflow better
> > for
> > > everyone.
> > >
> > > I think it would be great to collaborate with others including Jarek,
> > > Kaxil, Jens, Bolke, Amogh etc. who have already or would like to share
> > > ideas, to brainstorm tenets, major themes, scope of Airflow 3, along
> with
> > > prioritized list. Maybe we can have a dedicated sync, say next Thursday
> > > 8:30am PT, to discuss this further? I can volunteer to organize these
> > > meetings. Initial set of meetings will be to brainstorm to bring
> forward
> > a
> > > comprehensive and aligned proposal forward for the community.
> > >
> > > I'm excited about Airflow 3 and looking forward to shaping Airflow's
> > > future together!
> > >
> > > Shubham
> > >
> > >
> > > On 2024-05-10, 8:45 AM, "Pierre Jeambrun" <pierrejbrun@gmail.com
> > <mailto:
> > > pierrejbrun@gmail.com>> wrote:
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > > click links or open attachments unless you can confirm the sender and
> > know
> > > the content is safe.
> > >
> > >
> > >
> > >
> > >
> > >
> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe.
> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> > pouvez
> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> > que
> > > le contenu ne présente aucun risque.
> > >
> > >
> > >
> > >
> > >
> > >
> > > Thank you for the work and proposal made around airflow 3.
> > >
> > >
> > > I also strongly believe that those important changes need to happen for
> > > airflow to stay relevant and it would be the right moment to start
> > > discussing and planning for airflow 3. As mentioned some features are
> > hard
> > > to develop within airflow 2 with backward compatibility in mind:
> > > * I would love to see a full react front end, without FAB.
> > > * We could also take the opportunity to rework and clarify our RBAC
> code,
> > > we had quite a number of CVEs related to it.
> > > * Do the breaking change for api clients and (finally) upgrade the
> > openapi
> > > client generator to a more recent version
> > > * Extract business service layers that could be shared across the
> entire
> > > codebase instead of having a 'component wise' approach. Such that
> > handling
> > > of dag dates (execution, logical), serialization, resource access
> control
> > > and such tools could be implemented once and shared everywhere. (easier
> > > maintenance, more consistency between API, CLI, and other core
> components
> > > behaviors). Inspired by some famous software architecture such as
> > hexagonal
> > > or clean. Even if I don't think that those would be relevant to airflow
> > > codebase stricto sensu.
> > >
> > >
> > > As mentioned stability for our users is key and supporting airflow 2.x
> > for
> > > a while after airflow 3 is a must have. I just want to bring to
> > > your attention that this means more work for our release managers and
> > also
> > > for the security team (backporting security patches, CVEs handling for
> > two
> > > different major versions, etc.). All that to say that we should choose
> > this
> > > period wisely. (long enough, but not too long)
> > >
> > >
> > > This sounds like a really exciting time for airflow.
> > > Cheers
> > >
> > >
> > > Le jeu. 9 mai 2024 à 11:23, Amogh Desai <amoghdesai.oss@gmail.com
> > <mailto:
> > > amoghdesai.oss@gmail.com>> a écrit :
> > >
> > >
> > > > Thanks for initiating this discussion @Kaxil Naik <
> kaxil@astronomer.io
> > > <ma...@astronomer.io>>!
> > > >
> > > >
> > > > Read through the document and the emails here and I like the
> direction
> > in
> > > > which
> > > > we are proceeding.
> > > >
> > > > I also agree that removing code is almost as important as adding
> code.
> > We
> > > > need to have a clear checklist of what we want to
> > > > remove and get a vote for the same from the community.
> > > >
> > > > I also see that we have multiple ways of doing a thing, which is not
> > > needed
> > > > (serialising for example) and removing
> > > > which can lead to simpler interfaces for users, and contributors too.
> > > >
> > > > I also find the *secret masker* overly complex personally, why can't
> it
> > > be
> > > > simpler :)
> > > >
> > > > However, I have a small concern with the proportion of the users that
> > > would
> > > > like to migrate to an early version of
> > > > Airflow 3 if we do not support various deployment modes. If we dont
> get
> > > > enough initial feedback, we do not know how the new
> > > > major is doing, no?
> > > >
> > > > Thanks & Regards,
> > > > Amogh Desai
> > > >
> > > >
> > > > On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <ash@apache.org
> > > <ma...@apache.org>> wrote:
> > > >
> > > > > Strong agree - I think one of the things I regret most about
> Airflow
> > 2
> > > > was
> > > > > that we didn't remove enough. It probably eased the upgrade process
> > for
> > > > > users though.
> > > > >
> > > > > > but I was hoping one day when we remove providers
> > > > > > from main development, I can do it personally and beat Ash to it.
> > > > >
> > > > > Grrr ;l
> > > > >
> > > > > The other thing I want to look at (but don't have concrete ideas of
> > > yet)
> > > > > is some of the lesser used dag/task concurrency and dependency
> > > features -
> > > > > if we can remove a few things and get a faster scheduler than it's
> > > worth
> > > > > having a discussion over.
> > > > >
> > > > > On 9 May 2024 06:31:38 BST, Jarek Potiuk <jarek@potiuk.com
> <mailto:
> > > jarek@potiuk.com>> wrote:
> > > > > >I hope more voices will be coming soon as well :).
> > > > > >
> > > > > >Constance - few words of explanation, for my "remove" things. I do
> > not
> > > > say
> > > > > >we "have to" remove those things or even that we "should".
> > > > > >
> > > > > >My main motivation for mentioning the list is that we often forget
> > > that
> > > > > >when we have a new major version of software - removing stuff is
> as
> > > > > >important as adding. So I think when it comes to final decisions
> > about
> > > > > >Airflow 3 we should be well aware about both: what we add and what
> > we
> > > > > >remove. So my main point is - "removal" of things should be as
> > > > important a
> > > > > >part in our future discussions as "adding", and I'd say with 10
> > years
> > > of
> > > > > >Airflow history, it's more important to remove things than to add
> > > them.
> > > > > >
> > > > > >Side comment: Ash for one mentioned at multiple occasions how
> proud
> > he
> > > > is
> > > > > >that his contributions to Airflow are net-negative - he removed
> more
> > > > code
> > > > > >than added. Unfortunately it's not shown any more here
> > > > > >https://github.com/apache/airflow/graphs/contributors <
> > > https://github.com/apache/airflow/graphs/contributors> - because we
> have
> > > > > >more than 10.000 commits, but I was hoping one day when we remove
> > > > > providers
> > > > > >from main development, I can do it personally and beat Ash to it.
> > > > > >
> > > > > >Just wanted to stress a few things on why I think "discussing
> > > potential
> > > > > >removals" is important:
> > > > > >
> > > > > >* I think it is super important to deliver Airflow 3 really fast.
> It
> > > > took
> > > > > >
> > > > > >1.5 year to release Airflow 2 and it was far too long and keeping
> > > > Airflow
> > > > > 1
> > > > > >and 2 in parallel was a huge pain and drain and it slowed us down
> > > > > >enormously. We should absolutely limit the time when we actively
> > > develop
> > > > > >both Airflow 2 features and Airflow 3 features.
> > > > > >* I think most of the current deprecations of the 100+ we have
> does
> > > not
> > > > > >slow us down AT ALL. Removal of those is mostly cosmetic change, a
> > > > little
> > > > > >bit clutter to remove but they have little-to-no impact on actual
> > > speed
> > > > of
> > > > > >development, it's just an old code that keeps on being around
> > > > > >* on the other hand - some of the things I mentioned ARE slowing
> us
> > > > down A
> > > > > >LOT and will continue to do so until we remove them. For example,
> I
> > > > > believe
> > > > > >"postgresql + mysql + sqlite complete versioning solution for all
> > the
> > > > > >possible variants of storage" will be quite a bit more complex and
> > > > testing
> > > > > >will take far more time than if we drop mysql and choose a single
> > > > storage
> > > > > >approach for Airflow 3.0. I have no hard data to back it up, but
> my
> > > gut
> > > > > >feeling is that it can take at least twice as long to get it out
> in
> > > the
> > > > > >hands of our users if we try to have Airflow 2 parity in Airflow
> 3.0
> > > for
> > > > > >all the options we have there.
> > > > > >* the telemetry will be cool - but even if we add it for 2.10, the
> > > first
> > > > > >time meaningful data will be available is maybe 2 years from now
> > when
> > > we
> > > > > do
> > > > > >Airflow 4. For now we can completely forget about it because it's
> > only
> > > > > >going to show us some early adopters of Airflow 2.10. But we have
> > > > surveys
> > > > > +
> > > > > >we can collect data from Astronomer, Google. Amazon for some usage
> > and
> > > > > >identify who will be early adopters of Airflow 3 and target them
> > > first.
> > > > > >* the fact that we will drop something for Airflow 3.0, does not
> > mean
> > > > that
> > > > > >we have to drop it "forever". We can safely assume that Airflow
> 3.0
> > > will
> > > > > be
> > > > > >only used by early adopters, and many of our users will wait for
> > 3.1,
> > > > 3.2
> > > > > >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those
> > > > "slower
> > > > > >moving users" from the 3.0. We can re-add things in 3.1 building
> on
> > > > > >foundations of 3.0 after it is in the hands of those early
> > adopters. I
> > > > > >personally think we should prioritize speed of delivery of 3.0
> over
> > > > > >"complete support of every deployment option of Airflow 2" - to
> > allow
> > > > > >**some** of our early adopter users to migrate quickly, and add
> > what's
> > > > > >missing for those who would like to move later in later versions.
> > > > > >* I think - this is the most important "product management"
> decision
> > > > here
> > > > > -
> > > > > >based on the data we have available. Which users do we target for
> > > > Airflow
> > > > > >3.0, and what we can drop to deliver it faster (and possibly
> re-add
> > > more
> > > > > >stuff for 3.1+). I think it's crucial to the success of the
> Airflow
> > > 3.0
> > > > > >initiative and the most important decision from the Product
> > management
> > > > > side
> > > > > >that will impact the timeline of Airflow 3.0. While a lot depends
> on
> > > how
> > > > > >much time and effort individuals will spend on implementing
> things,
> > > the
> > > > > >decision of what we drop will impact everyone's speed and overall
> > > > delivery
> > > > > >of Airflow 3.0. It's a decision we should not take lightly.
> > > > > >
> > > > > >So my main point is: let's add "what we remove" as a very
> important
> > > > point
> > > > > >in the product discussions we are going to have.
> > > > > >
> > > > > >J.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar
> > > > > ><rbishund@amazon.com.inva <ma...@amazon.com.inva>lid>
> > > wrote:
> > > > > >
> > > > > >> Like Constance and many others, there was time needed to process
> > > this
> > > > > >> fantastical summary and approach to Airflow 3. __
> > > > > >> A lot of the points raised by Jarek make sense, ie being
> > > prescriptive
> > > > on
> > > > > >> what features we want to be there on Day 1 vs launched on 3.x -
> > this
> > > > > does
> > > > > >> allow us to move quickly for our users.
> > > > > >> In line with Constance's concern, we need to be mindful of what
> we
> > > > > decide
> > > > > >> to keep and what we are willing to cut - which could be a
> > > painstaking
> > > > > >> process that could offset any gains of trying to be faster. I
> also
> > > > > believe
> > > > > >> that bringing all these new amazing features on Airflow 3 will
> > peak
> > > > the
> > > > > >> interest of early adopters and eventually get others interested
> in
> > > > > >> migration. However, I believe this migration will be a slow
> > process
> > > > and
> > > > > >> will present a gap in certain functionalities that users may
> want
> > > > before
> > > > > >> entertaining any move to Airflow 3. There are still a lot of
> folks
> > > > using
> > > > > >> v1.10 today. There were several tactical initiatives in the past
> > few
> > > > > months
> > > > > >> with intent on bringing new functionality, ie Multi-team, to
> > Airflow
> > > > 2.x
> > > > > >> and I feel that while these efforts are not wasted, there should
> > > still
> > > > > be
> > > > > >> an option to continue improving Airflow 2 to avoid alienating
> our
> > > > users
> > > > > on
> > > > > >> the basis of a future promise in Airflow 3, that may not be easy
> > to
> > > > > migrate
> > > > > >> towards.
> > > > > >>
> > > > > >> -- Rajesh
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > > dev@airflow.apache.org <ma...@airflow.apache.org>> <
> > > > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > > dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > > > > >>
> > > > > >>
> > > > > >> On 2024-05-07, 12:26 PM, "Constance Martineau"
> > > > > >> <constance@astronomer.io.INVA <mailto:
> > constance@astronomer.io.INVA>
> > > <mailto:constance@astronomer.io.INVA <mailto:
> > constance@astronomer.io.INVA
> > > >>
> > > > > >> <mailto:constance@astronomer.io.INVA <mailto:
> > > constance@astronomer.io.INVA> <mailto:
> > > > > constance@astronomer.io.INVA <mailto:constance@astronomer.io.INVA
> > > >>>LID>
> > > > > >> wrote:
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> CAUTION: This email originated from outside of the organization.
> > Do
> > > > not
> > > > > >> click links or open attachments unless you can confirm the
> sender
> > > and
> > > > > know
> > > > > >> the content is safe.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > > > > externe.
> > > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> vous
> > ne
> > > > > pouvez
> > > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > > certain
> > > > > que
> > > > > >> le contenu ne présente aucun risque.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> Thank you Jarek for the detailed input. I've taken some time to
> > > digest
> > > > > your
> > > > > >> points before responding.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> You've outlined a bold vision for Airflow 3, and I agree that
> > being
> > > > > >> decisive about the features and architectures will be the key to
> > > > > success.
> > > > > >> However, before we make final decisions on what features to cut
> or
> > > > > retain,
> > > > > >> it would be beneficial to have a more comprehensive
> understanding
> > of
> > > > how
> > > > > >> the current features are utilized by the open-source community.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> @Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io>
> > > <mailto:kaxil@astronomer.io <ma...@astronomer.io>>
> > > > <mailto:
> > > > > >> kaxil@astronomer.io <ma...@astronomer.io> <mailto:
> > > kaxil@astronomer.io <ma...@astronomer.io>>>> recently
> > > > initiated a
> > > > > >> discussion on
> > > > > >> collecting telemetry from open-source deployments:
> > > > > >>
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m
> > <
> > > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> > > > > >>
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>
> > <
> > > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;>
> <
> > > > > >>
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>
> > <
> > > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;>
> <
> > > > > >>
> > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt
> > > <
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&amp;gt
> > >
> > > > ;>.
> > > > > >> This data
> > > > > >> could be critical in ensuring our decisions are well-informed
> and
> > > > > reflect
> > > > > >> the real-world usage patterns of our users, not just those from
> > > > managed
> > > > > >> environments like Astro, MWAA or GCC.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> It's essential that we challenge our assumptions and base our
> > > > decisions
> > > > > on
> > > > > >> a holistic view of feature usage. Identifying potential cuts is
> a
> > > > > critical
> > > > > >> step, but let's ensure our strategy aligns with the needs and
> > > > > preferences
> > > > > >> of the broader Airflow community.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <jarek@potiuk.com
> > > <ma...@potiuk.com>
> > > > <mailto:
> > > > > >> jarek@potiuk.com <ma...@potiuk.com>> <mailto:
> > > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com
> > > <ma...@potiuk.com>
> > > > >>>
> > > > > >> wrote:
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> > I am currently on sick leave, and still recovering - hoping to
> > be
> > > > > able to
> > > > > >> > travel next week to the US as planned, so I just wanted to
> break
> > > out
> > > > > of
> > > > > >> it
> > > > > >> > to make one comment here.
> > > > > >> >
> > > > > >> > I got a clearer head now a bit with medications hopefully
> > > working. I
> > > > > am
> > > > > >> > still taking it that should help me to get over the current
> > state,
> > > > > and I
> > > > > >> > wanted to take a look at this discussion unraveling first.
> Over
> > > last
> > > > > week
> > > > > >> > I disconnected from "day-to-day" Airflow and put some thoughts
> > (as
> > > > > much
> > > > > >> as
> > > > > >> > I could in my current state) on it. The whole subject of this
> > > thread
> > > > > was
> > > > > >> > started from that - how the current discussions on AIP-67 and
> > > others
> > > > > >> change
> > > > > >> > if we consider Airflow 3 is "starting".
> > > > > >> >
> > > > > >> > The price for back-compat is speed of development and quality.
> > > More
> > > > > >> > combinations to test, more unexpected issues uncovered,
> > necessity
> > > to
> > > > > keep
> > > > > >> > parallel paths (old/new) while adding new features. All what
> > > > Constance
> > > > > >> > wrote about and what Ash explained. We already started to trip
> > > over
> > > > > our
> > > > > >> own
> > > > > >> > feet mutliple times in a few last releases. Have we tested all
> > > > > >> combinations
> > > > > >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we
> > > > already
> > > > > see
> > > > > >> > that in a number of "combos" of features things are not
> working
> > in
> > > > as
> > > > > >> > stable a way as they did before.
> > > > > >> >
> > > > > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2
> > > for a
> > > > > long
> > > > > >> > time (or even move out) as they will not want to move to
> Airflow
> > > 3.
> > > > A
> > > > > lot
> > > > > >> > of the work implemented in AIP-44 and design of AIP-67 was
> done
> > > > around
> > > > > >> > back-compatibility. but yes -
> > > > > >> > it would have been way easier if designed anew without
> > > > > back-compatibility
> > > > > >> > in mind. And if we implement it and release it in Airflow 2 it
> > > will
> > > > > make
> > > > > >> > new Airflow feature development even harder. That's why I
> wanted
> > > to
> > > > > treat
> > > > > >> > it as "tactical" solution - hoping that in Airflow 3 we can
> make
> > > it
> > > > > >> > "properly" - and that's why I started the discussion here
> when I
> > > > > sensed
> > > > > >> > that we are "close" to Airflow 3 discussion, because I wanted
> to
> > > see
> > > > > what
> > > > > >> > options we have there. This is why I have not yet concluded
> > voting
> > > > on
> > > > > >> > AIP-67 waiting for the result of this discussion here.
> > > > > >> >
> > > > > >> > But if we are ready to go for Airflow.3 then I'd say there are
> > two
> > > > > >> > important things that should be part of the vision.
> > > > > >> >
> > > > > >> > 1) *We should be far more opinionated and have far fewer
> options
> > > of
> > > > > >> > running things in Airflow 3*. Even an order of magnitude more
> > > > > >> opinionated.
> > > > > >> > Make choices, stick to it, perfect those opinionated choices
> to
> > > suit
> > > > > >> 80/20
> > > > > >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking
> > not
> > > > > fitting
> > > > > >> > the 20% that might choose to stay at Airflow 2. We can choose
> > now
> > > > > which
> > > > > >> > ~20% of cases we do not want to handle deliberately. And we
> > should
> > > > be
> > > > > >> very,
> > > > > >> > very strict about it. Default should be "no choice". This will
> > > > > radically
> > > > > >> > simplify deployment and should make it easier to simplify
> > Airflow
> > > > > >> > development and DAG authoring experience because we will have
> > less
> > > > > cases
> > > > > >> to
> > > > > >> > support. Even if we plan to add more options in the future,
> the
> > > > first
> > > > > >> > version of Airflow 3 should support one deployment approach
> > only.
> > > > > This is
> > > > > >> > the only way we can deliver it fast. And we should be very
> bold
> > > > there.
> > > > > >> > Choose one option and go for it in pretty much every place we
> > have
> > > > > >> choices
> > > > > >> > now. We should Aim for Airflow 3.0 to support only a subset of
> > > > current
> > > > > >> > users - but those who are most likely to migrate first and
> those
> > > > with
> > > > > the
> > > > > >> > biggest need for the new features. We can think 3.x to support
> > > more
> > > > > >> cases,
> > > > > >> > but 3.0 should be as opinionated as humanly possible.
> > > > > >> >
> > > > > >> > And this deployment option should be also something ALL our
> > > > > stakeholders
> > > > > >> > will feel OK with as a way forward in their offering.
> > > > > >> >
> > > > > >> > My candidates (and yes, some are bold):
> > > > > >> >
> > > > > >> > * *Drop MySQL*. If we have a single thing that makes us avoid
> > our
> > > > > schema
> > > > > >> > and DB migration - this is the case. Let's choose Postgres 15+
> > and
> > > > use
> > > > > >> some
> > > > > >> > of the great features there. This will also enable much faster
> > > async
> > > > > SQL
> > > > > >> > implementation and a number of other optimisations - not to
> > > mention
> > > > > >> cutting
> > > > > >> > every single change in development and testing time by
> literally
> > > > half.
> > > > > >> And
> > > > > >> > we should not look back to adding MySQL.
> > > > > >> > * *Drop Celery/Sequential Executor* and start with Local + K8S
> > > only
> > > > > (and
> > > > > >> > AWS/Google others can continue developing theirs of course in
> > > > parallel
> > > > > >> and
> > > > > >> > continue Hybrid executor work). Later - we figure out a better
> > > > > solution
> > > > > >> to
> > > > > >> > support "small" tasks using some new K8S features and possibly
> > > > non-k8s
> > > > > >> > solutions (Ray-based?)
> > > > > >> > * *Cut Connection and Variable Management from DB/UI*. Leave
> > only
> > > > > Secrets
> > > > > >> > Management. Later when we have a 100% extensible React UI, we
> > can
> > > > add
> > > > > a
> > > > > >> > "local DB secrets manager" add-on
> > > > > >> > * *Choose a single way for DAG storage that will support
> > > versioning
> > > > > from
> > > > > >> > day one*. Bear in mind we can add others later. Bolke's idea
> of
> > > > using
> > > > > >> > FSspec is an interesting one, we should see if it is feasible.
> > > > > >> > * *Drop FAB completely (including custom plugins) and invest
> in
> > > > > >> > implementing Auth Manager based on a dedicated, external
> > solution*
> > > > > >> > (KeyCloak
> > > > > >> > that we've discussed before as a likely candidate)
> > > > > >> > * *Leave Providers with Airflow 2 and add tests to make sure
> > they
> > > > are
> > > > > >> > Airflow 3 future-compatible *- develop a way where we continue
> > > > > >> development
> > > > > >> > and contributions for Providers with Airflow 2 and add
> complete
> > > > tests
> > > > > to
> > > > > >> > run them with Airflow 3. This way we can continue developing
> > > > Provider
> > > > > >> > features independently, and make them work for Airflow 2 (and
> > > > continue
> > > > > >> > adding features for Airflow 2 users alongside Airflow 2
> > bugfixes),
> > > > > while
> > > > > >> > also gradually fix any Airflow3 incompatibilities and instead
> of
> > > > > >> > "back-compatibility" tests make provider
> "forward-compatibility"
> > > > > tests so
> > > > > >> > that future Providers are tested and work on Airflow 3. Also
> it
> > > will
> > > > > make
> > > > > >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested
> > > > without
> > > > > >> > investing in changing the current CI / test harness.
> > > > > >> > * *Simplify Test Harness for Airflow 3 from the start *-
> without
> > > > > >> providers
> > > > > >> > and 790+ dependencies, we could vastly simplify Airflow3
> testing
> > > > > >> (basically
> > > > > >> > make CI jobs from scratch) using mostly standard Python
> tooling
> > > > > (while we
> > > > > >> > can continue making use of the current test harness for
> Airflow
> > 2
> > > +
> > > > > >> > Providers and extend it with Airflow 3 future-compatibility
> > > tests).
> > > > > That
> > > > > >> > means Breeze would be only staying in Airflow 2 + Providers
> repo
> > > as
> > > > we
> > > > > >> > should be able to achieve most of what we have there with
> local
> > > > venv/
> > > > > >> > tooling (especially with uv as underlying tooling).
> > > > > >> >
> > > > > >> > 2) *I think we only add very few new "important" features.
> > > *Absolute
> > > > > >> > minimum to make Airflow 3 appealing and add them only in
> Airflow
> > > 3:
> > > > > >> > versioning, multi-team, pluggable UI should only be Airflow 3
> -
> > it
> > > > > makes
> > > > > >> no
> > > > > >> > sense to invest into Airflow 2 if we already know Airflow 3 is
> > > > coming
> > > > > -
> > > > > >> > that generally triples effort needed to get them out. We
> should
> > > drop
> > > > > new
> > > > > >> > features development in Airflow 2. This will give users
> > incentive
> > > to
> > > > > move
> > > > > >> > to 3 if the new features will be worth it. Even paying
> > > > > >> > compatibility/migration price.
> > > > > >> >
> > > > > >> > Versionig, for example: I believe if we decide to go only with
> > > > > Airflow 3
> > > > > >> > and cut some of the above (Postgres only, Single versioning
> DAG
> > > > > storage)
> > > > > >> we
> > > > > >> > can make bolder decisions in versioning and support simpler
> > models
> > > > > from
> > > > > >> the
> > > > > >> > get go (and deliver it faster). And we should add only a few -
> > but
> > > > > >> > important - features that our users clearly asked for and
> focus
> > on
> > > > > >> > delivering Airflow 3 as soon as possible (instead of Airflow
> > 2.10
> > > or
> > > > > >> 2.11).
> > > > > >> > Similarly - multi-team can be simplified if we cut things from
> > the
> > > > > list
> > > > > >> > above and have Task isolation as first-class citizens in
> Airflow
> > > > (and
> > > > > the
> > > > > >> > only option).
> > > > > >> >
> > > > > >> > My candidates very much concur with the list shared by Kaxil
> in
> > > the
> > > > > doc +
> > > > > >> > I'd add multi-team (but simplified thanks to the cuts). But I
> > also
> > > > > here
> > > > > >> > would mostly revert to Astronomer, Google. AWS team to define
> > > > > >> collectively
> > > > > >> > what is the absolute minimum set of features that would get
> the
> > > > > "target"
> > > > > >> > part of their customers happy. And ONLY do that.
> > > > > >> >
> > > > > >> > So in short - I think the big part of our discussion should be
> > > what
> > > > we
> > > > > >> are
> > > > > >> > ready to drop when we start airflow 3 and be very bold. Once
> we
> > > know
> > > > > we
> > > > > >> > should figure out the absolute minimum of things that we can
> add
> > > > that
> > > > > >> will
> > > > > >> > benefit a significant part of our users (and make use of
> > increased
> > > > > speed
> > > > > >> > because we dropped things).
> > > > > >> >
> > > > > >> > J.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau
> > > > > >> > <constance@astronomer.io.inva <mailto:
> > > constance@astronomer.io.inva> <mailto:constance@astronomer.io.inva
> > > <ma...@astronomer.io.inva>>
> > > > > >> <mailto:constance@astronomer.io.inva <mailto:
> > > constance@astronomer.io.inva> <mailto:
> > > > > constance@astronomer.io.inva <mailto:constance@astronomer.io.inva
> > > >>>lid>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Hi Michal,
> > > > > >> > >
> > > > > >> > > Thanks for your thoughts on the Airflow 3 proposal. I
> > appreciate
> > > > > your
> > > > > >> > > concerns about the migration overhead for our users with a
> > major
> > > > new
> > > > > >> > > version and see the appeal in your suggestion to integrate
> > many
> > > of
> > > > > the
> > > > > >> > > proposed changes into Airflow 2 through separate AIPs. It’s
> a
> > > > valid
> > > > > >> point
> > > > > >> > > and certainly aligns with the value of making incremental
> > > > > improvements.
> > > > > >> > >
> > > > > >> > > However, after looking closely at the enhancements outlined
> > for
> > > > > Airflow
> > > > > >> > 3,
> > > > > >> > > I'm convinced they warrant a new major release. Here’s why:
> > > > > >> > >
> > > > > >> > > 1. *Core Architectural Changes:* We’re looking at
> foundational
> > > > > changes
> > > > > >> > > with Airflow 3—like redefining task priorities, separating
> > task
> > > > > >> > > definition
> > > > > >> > > and task execution, and new AIPs like DAG versioning. remote
> > > > > execution
> > > > > >> > > and restricting database access from workers. These aren’t
> > just
> > > > > >> > > incremental
> > > > > >> > > improvements but major shifts that will set the stage for
> the
> > > next
> > > > > >> > > decade
> > > > > >> > > of Airflow’s architecture. Grouping these changes into a
> major
> > > > > release
> > > > > >> > > will
> > > > > >> > > help us make these transitions more cleanly and with fewer
> > > > > constraints
> > > > > >> > > from
> > > > > >> > > past decisions.
> > > > > >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140
> > > > > >> > deprecated
> > > > > >> > > issues, and this will only grow if we continue without a
> major
> > > > > >> > cleanup.
> > > > > >> > > This makes it increasingly difficult to implement new
> features
> > > > > >> > > effectively
> > > > > >> > > while maintaining backward compatibility. A major release
> > allows
> > > > us
> > > > > to
> > > > > >> > > address these issues head-on, reducing technical debt and
> > paving
> > > > the
> > > > > >> > way
> > > > > >> > > for a more robust platform.
> > > > > >> > > 3. *Managing Breaking Changes:* Let’s take the example of
> > > > > restricting
> > > > > >> > > database access from workers. It’s a necessary move for
> better
> > > > > >> > security
> > > > > >> > > and
> > > > > >> > > also potentially scalability reasons (reduces DB load). Many
> > > users
> > > > > >> > have
> > > > > >> > > workflows that interact with the DB, either by using raw sql
> > or
> > > by
> > > > > >> > > leveraging a session object. We could implement this feature
> > in
> > > > > >> > Airflow
> > > > > >> > > 2
> > > > > >> > > and avoid breaking existing workflows by continuing to have
> > the
> > > > old
> > > > > >> > > standard mode as default - much of the work is already done
> -
> > > but
> > > > > that
> > > > > >> > > would mean supporting both the new secure mode and the old
> > > > standard
> > > > > >> > mode
> > > > > >> > > indefinitely and design new features with the assumption
> that
> > > most
> > > > > >> > will
> > > > > >> > > continue using the old standard mode. With Airflow 3, we can
> > > make
> > > > > >> > secure
> > > > > >> > > mode the default or even the only option, simplifying
> > > > implementation
> > > > > >> > and
> > > > > >> > > future development. This is just one example where it is
> > > feasible
> > > > to
> > > > > >> > > implement in Airflow 2, but is better if we release it under
> > the
> > > > > >> > > context of
> > > > > >> > > Airflow 3.
> > > > > >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open
> up
> > > > > >> > > possibilities for handling workflows beyond batch
> processing.
> > > > > Features
> > > > > >> > > like
> > > > > >> > > real-time DAG execution through API and multi-language task
> > > > support
> > > > > >> > are
> > > > > >> > > big
> > > > > >> > > steps forward, significantly expanding Airflow’s utility.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > While integrating these updates into Airflow 2 might look
> less
> > > > > >> disruptive
> > > > > >> > > initially, the scale and nature of the required changes
> really
> > > > > support
> > > > > >> a
> > > > > >> > > move to Airflow 3. It’s not just about adding new features;
> > it’s
> > > > > about
> > > > > >> > > setting up Airflow so that it continues to remain relevant
> for
> > > the
> > > > > next
> > > > > >> > ten
> > > > > >> > > years.
> > > > > >> > >
> > > > > >> > > Constance
> > > > > >> > >
> > > > > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <
> > > ash@apache.org <ma...@apache.org>
> > > > > >> <mailto:ash@apache.org <ma...@apache.org>> <mailto:
> > > ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <mailto:
> > > ash@apache.org>
> > > > > >>>
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > > There's a lot of technical debt hiding in Airflow,
> > especially
> > > > the
> > > > > >> > > > scheduler that makes it harder and harder to efficiently
> add
> > > new
> > > > > >> > > features.
> > > > > >> > > >
> > > > > >> > > > At some point, very soon, we are going to have to remove
> > some
> > > > very
> > > > > >> > > > infrequently used back compat shims that negatively affect
> > > > > >> performance.
> > > > > >> > > > Without doing that the pace at which we can realistically
> > add
> > > > > some of
> > > > > >> > the
> > > > > >> > > > more exciting features tends towards zero. Developer speed
> > of
> > > > > >> > > contributors
> > > > > >> > > > is a factor here too!
> > > > > >> > > >
> > > > > >> > > > So while we are still using SemVer, that necessitates v3.
> > > > > >> > > >
> > > > > >> > > > Ash
> > > > > >> > > >
> > > > > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <
> > > > > michalmodras@google.com <ma...@google.com>
> > > > > >> <mailto:michalmodras@google.com <mailto:michalmodras@google.com
> >>
> > > <mailto:michalmodras@google.com <ma...@google.com>
> > > > > <mailto:
> > > > > >> michalmodras@google.com <ma...@google.com>>>
> > > > > >> > > .INVALID>
> > > > > >> > > > wrote:
> > > > > >> > > > >+1 to Jens's & Bolke's points here and in the doc
> > > > > >> > > > >
> > > > > >> > > > >I agree we should work on clarifying the directions we
> > would
> > > > like
> > > > > >> > > Airflow
> > > > > >> > > > >to go. Introducing a new major Airflow version is a
> massive
> > > > > overhead
> > > > > >> > for
> > > > > >> > > > >users, who would need to plan for migrations, onboarding
> > the
> > > > new
> > > > > >> > Airflow
> > > > > >> > > > >(with a slightly different architecture), etc., and
> > > effectively
> > > > > >> > Airflow
> > > > > >> > > 2
> > > > > >> > > > >would live in parallel for a long time.
> > > > > >> > > > >
> > > > > >> > > > >Personally, I think most of the points in
> Kaxil's/Vikram's
> > > doc
> > > > > are
> > > > > >> > > > valuable
> > > > > >> > > > >projects of their own, and I could imagine all of them
> > being
> > > > > >> delivered
> > > > > >> > > as
> > > > > >> > > > >separate AIPs within Airflow 2 (surely new minor versions
> > of
> > > > > Airflow
> > > > > >> > > 2). I
> > > > > >> > > > >am not sure if the scope of changes and the goal we want
> to
> > > > > achieve
> > > > > >> is
> > > > > >> > > a)
> > > > > >> > > > >clear enough b) broad enough to call for a new major
> > version.
> > > > > >> > > > >
> > > > > >> > > > >Best,
> > > > > >> > > > >Michal
> > > > > >> > > > >
> > > > > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens
> > > > (XC-AS/EAE-ADA-T)
> > > > > >> > > > ><Jens.Scheffler@de.bosch.com.inva <mailto:
> > > Jens.Scheffler@de.bosch.com.inva> <mailto:
> > > > > >> Jens.Scheffler@de.bosch.com.inva <mailto:
> > > Jens.Scheffler@de.bosch.com.inva>> <mailto:
> > > > > Jens.Scheffler@de.bosch.com.inva <mailto:
> > > Jens.Scheffler@de.bosch.com.inva>
> > > > > >> <mailto:Jens.Scheffler@de.bosch.com.inva <mailto:
> > > Jens.Scheffler@de.bosch.com.inva>>>lid> wrote:
> > > > > >> > > > >
> > > > > >> > > > >> Thanks for the document write-up, Kaxil. I assume this
> is
> > > > > mostly a
> > > > > >> > > > vision
> > > > > >> > > > >> statement.
> > > > > >> > > > >>
> > > > > >> > > > >> Looking forward for a larger addendum where we can
> > collect
> > > > > things
> > > > > >> > that
> > > > > >> > > > we
> > > > > >> > > > >> all can vote and agree on as targets.
> > > > > >> > > > >>
> > > > > >> > > > >> As I started earlier with a confluence page and it
> seems
> > > this
> > > > > is
> > > > > >> not
> > > > > >> > > > >> accessible to all, shall we convert this to a Google
> Doc
> > > for
> > > > > >> better
> > > > > >> > > > >> collaboration and item collection?
> > > > > >> > > > >>
> > > > > >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> <
> > > https://aka.ms/o0ukef&gt;> <
> > > > > >> https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&amp;gt;&gt;>
> <
> > > https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&amp;gt;&gt;> <
> > > > > >> https://aka.ms/o0ukef&amp;gt;&gt;> <
> > > https://aka.ms/o0ukef&amp;amp;gt;&amp;gt;&gt;>
> > > > > >> > > > >> ________________________________
> > > > > >> > > > >> From: Vikram Koka <vikram@astronomer.io.INVA <mailto:
> > > vikram@astronomer.io.INVA> <mailto:
> > > > > >> vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>>
> > > <mailto:vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>
> > > <mailto:
> > > > > >> vikram@astronomer.io.INVA <mailto:vikram@astronomer.io.INVA
> > >>>LID>
> > > > > >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> > > > > >> > > > >> To: dev@airflow.apache.org <mailto:
> > dev@airflow.apache.org>
> > > <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>
> > > > > >> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>
> > > <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>> <
> > > > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > > dev@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > > > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > > dev@airflow.apache.org <ma...@airflow.apache.org>>>>
> > > > > >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical
> > > (Airflow
> > > > > 2)
> > > > > >> vs
> > > > > >> > > > >> strategic (Airflow 3) approach
> > > > > >> > > > >>
> > > > > >> > > > >> Thank you for your feedback, Bolke and Andrey!
> > > > > >> > > > >>
> > > > > >> > > > >> Bolke,
> > > > > >> > > > >> I have replied to some of your comments in the doc.
> > > > > >> > > > >> I will provide a detailed write up on the "Interactive
> > DAG
> > > > run"
> > > > > >> (or
> > > > > >> > > > >> synchronous DAG run) capability, which has generated
> some
> > > > early
> > > > > >> > > > questions.
> > > > > >> > > > >> I had intended to get an AIP published for that as a
> > > > follow-up,
> > > > > >> but
> > > > > >> > I
> > > > > >> > > > >> believe that a simpler write up would be useful ahead
> of
> > > the
> > > > > AIP.
> > > > > >> > > > >>
> > > > > >> > > > >> Andrey,
> > > > > >> > > > >> You raise an interesting point.
> > > > > >> > > > >>
> > > > > >> > > > >> As part of the Airflow 2.0 release, we as a community
> had
> > > > > decided
> > > > > >> to
> > > > > >> > > > >> strictly adhere to Semver as detailed in the document
> you
> > > > > >> > referenced.
> > > > > >> > > We
> > > > > >> > > > >> also consciously split out the "Core Airflow" releases
> > from
> > > > the
> > > > > >> > > > "Provider"
> > > > > >> > > > >> releases at that time. We had a clear expectation then
> > for
> > > > the
> > > > > >> > cadence
> > > > > >> > > > of
> > > > > >> > > > >> both minor and patch releases, which we have generally
> > > > adhered
> > > > > to
> > > > > >> > > since
> > > > > >> > > > >> then.
> > > > > >> > > > >>
> > > > > >> > > > >> Personally, I am more concerned about our Provider
> > releases
> > > > > right
> > > > > >> > now,
> > > > > >> > > > as
> > > > > >> > > > >> compared to the cadence of our major releases. I
> believe
> > > that
> > > > > one
> > > > > >> of
> > > > > >> > > the
> > > > > >> > > > >> proposed changes in the Airflow 3 document i.e. the
> clear
> > > > > >> separation
> > > > > >> > > for
> > > > > >> > > > >> Task Execution will help here, but more may be needed.
> > > > > >> > > > >>
> > > > > >> > > > >> Definitely interested in more feedback on this as well.
> > > > > >> > > > >>
> > > > > >> > > > >> Vikram
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> > > > > >> > > andrey.anshin@taragol.is <ma...@taragol.is>
> > > <mailto:andrey.anshin@taragol.is <ma...@taragol.is>>
> > > > <mailto:
> > > > > >> andrey.anshin@taragol.is <ma...@taragol.is>
> > <mailto:
> > > andrey.anshin@taragol.is <ma...@taragol.is>>>
> > > > > >> > > > >
> > > > > >> > > > >> wrote:
> > > > > >> > > > >>
> > > > > >> > > > >> > I would like to propose to change (at least discuss)
> > > > release
> > > > > >> > policy
> > > > > >> > > > >> around
> > > > > >> > > > >> > the Major version of Airflow.
> > > > > >> > > > >> >
> > > > > >> > > > >> > Right now it is described as "These releases do not
> > > happen
> > > > > with
> > > > > >> > any
> > > > > >> > > > >> regular
> > > > > >> > > > >> > interval or on any predictable schedule." :
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> > > <
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> > > >
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> > > <
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0
> > > >
> > > > > >
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> > > <
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0
> > > >
> > > > > >
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt
> > > <
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;amp;reserved=0&amp;gt
> > > >
> > > > > >> ;>
> > > > > >> > > > >> <
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > > >
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > > >
> > > > > >
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > > >
> > > > > >
> > > > > >> <
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&amp;gt
> > > >
> > > > > >> ;>
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >> > So maybe it is time to make it schedulable, e.g. one
> > per
> > > > two
> > > > > >> years
> > > > > >> > > or
> > > > > >> > > > so.
> > > > > >> > > > >> > This one could help us to avoid such a discussion in
> > the
> > > > > future,
> > > > > >> > > like
> > > > > >> > > > "We
> > > > > >> > > > >> > don't know when Airflow 4 is coming.". At the moment
> > when
> > > > the
> > > > > >> new
> > > > > >> > > > major
> > > > > >> > > > >> > version will be released new features wouldn't be
> added
> > > in
> > > > > the
> > > > > >> old
> > > > > >> > > > major
> > > > > >> > > > >> > version, however we would support bug / security for
> a
> > > > while,
> > > > > >> > e.g. 1
> > > > > >> > > > year
> > > > > >> > > > >> > for bug fixes, 3 years for security fixes with a
> total
> > 5
> > > > year
> > > > > >> > > > lifecycle
> > > > > >> > > > >> per
> > > > > >> > > > >> > a major version. These just are approximate time
> > periods
> > > > for
> > > > > a
> > > > > >> > > > definition
> > > > > >> > > > >> > of current period, bugfix period and security fix
> > period.
> > > > > >> > > > >> >
> > > > > >> > > > >> > In contributors' perspective it helps with dropping
> the
> > > > > >> deprecated
> > > > > >> > > > stuff
> > > > > >> > > > >> > which resolves some old problem: we have to support
> > > > > everything
> > > > > >> > > > including
> > > > > >> > > > >> > deprecated stuff and without schedulable lifecycle
> for
> > > the
> > > > > >> > > deprecated
> > > > > >> > > > >> stuff
> > > > > >> > > > >> > it could be showstopper for the new feature, because
> > > > > sometimes
> > > > > >> it
> > > > > >> > > > hard to
> > > > > >> > > > >> > support two different approaches for long period of
> > time
> > > > > with no
> > > > > >> > > hope
> > > > > >> > > > >> that
> > > > > >> > > > >> > it will happen soon. For some fundamental stuff which
> > do
> > > > not
> > > > > >> > > require a
> > > > > >> > > > >> lot
> > > > > >> > > > >> > things time to support we could postponed removal for
> > > next
> > > > > after
> > > > > >> > the
> > > > > >> > > > next
> > > > > >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it
> in
> > > > > Airflow 5
> > > > > >> > > > >> >
> > > > > >> > > > >> > In the user perspective, they have at least bug fix
> > > support
> > > > > for
> > > > > >> a
> > > > > >> > > > while,
> > > > > >> > > > >> if
> > > > > >> > > > >> > someone want to use legacy version it their choice,
> > > however
> > > > > no
> > > > > >> new
> > > > > >> > > > >> > features, no new version of providers (after one
> year)
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >> > ----
> > > > > >> > > > >> > Best Wishes
> > > > > >> > > > >> > *Andrey Anshin*
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <
> > > > > bdbruin@gmail.com <ma...@gmail.com>
> > > > > >> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> > > bdbruin@gmail.com <ma...@gmail.com> <mailto:
> > > > > >> bdbruin@gmail.com <ma...@gmail.com>>>>
> > > > > >> > > > wrote:
> > > > > >> > > > >> >
> > > > > >> > > > >> > > I have left several comments :-). And on
> interactive
> > > dag
> > > > > runs
> > > > > >> > even
> > > > > >> > > > >> after
> > > > > >> > > > >> > > the explanation of Vikram I still don't have a clue
> > > what
> > > > we
> > > > > >> want
> > > > > >> > > to
> > > > > >> > > > >> > > accomplish there :-P.
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > I would like to see a mantra or team for Airflow 3.
> > > That
> > > > > helps
> > > > > >> > > > nudging
> > > > > >> > > > >> > > people in the same direction. Suggestions in the
> > > > comments.
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > Bolke
> > > > > >> > > > >> > > Sent from my iPhone
> > > > > >> > > > >> > >
> > > > > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> > > > > >> > > <vikram@astronomer.io.inva <mailto:
> vikram@astronomer.io.inva>
> > > <mailto:vikram@astronomer.io.inva <ma...@astronomer.io.inva>>
> > > > > <mailto:
> > > > > >> vikram@astronomer.io.inva <ma...@astronomer.io.inva>
> > > <mailto:vikram@astronomer.io.inva <mailto:vikram@astronomer.io.inva
> > >>>lid
> > > > > >> > > > >
> > > > > >> > > > >> > > wrote:
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > > > Good point Jed.
> > > > > >> > > > >> > > > I responded back to your comment in the doc as
> well
> > > and
> > > > > very
> > > > > >> > > open
> > > > > >> > > > to
> > > > > >> > > > >> > > > changing the term in the doc.
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > > > Used the term "interactive DAG run" as the
> ability
> > to
> > > > > invoke
> > > > > >> > or
> > > > > >> > > > >> > trigger a
> > > > > >> > > > >> > > > DAG run through the API, with the expectation of
> > > > getting
> > > > > >> back
> > > > > >> > a
> > > > > >> > > > >> result
> > > > > >> > > > >> > > > immediately. An alternate term could be a
> > > "synchronous
> > > > > DAG
> > > > > >> > run".
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > > > Regardless, this is a significant change so a
> good
> > > term
> > > > > to
> > > > > >> > > > indicate
> > > > > >> > > > >> the
> > > > > >> > > > >> > > > expansion from "batch runs only" is warranted.
> Very
> > > > open
> > > > > to
> > > > > >> > > > different
> > > > > >> > > > >> > > terms
> > > > > >> > > > >> > > > here.
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > > > > >> > > > >> > jedcunningham@apache.org <mailto:
> > > jedcunningham@apache.org> <mailto:jedcunningham@apache.org <mailto:
> > > jedcunningham@apache.org>>
> > > > > >> <mailto:jedcunningham@apache.org <mailto:
> jedcunningham@apache.org
> > >
> > > <mailto:jedcunningham@apache.org <ma...@apache.org>>>
> > > > > >> > > > >> > > >
> > > > > >> > > > >> > > >> wrote:
> > > > > >> > > > >> > > >>
> > > > > >> > > > >> > > >> Very exciting! Looks like we will have a busy
> > period
> > > > of
> > > > > >> time
> > > > > >> > > > ahead
> > > > > >> > > > >> of
> > > > > >> > > > >> > > us.
> > > > > >> > > > >> > > >> Overall I like the plan so far, especially using
> > > this
> > > > > >> year's
> > > > > >> > > > Airflow
> > > > > >> > > > >> > > Summit
> > > > > >> > > > >> > > >> as an opportunity to announce and gather
> feedback,
> > > and
> > > > > the
> > > > > >> > 2025
> > > > > >> > > > >> > version
> > > > > >> > > > >> > > to
> > > > > >> > > > >> > > >> pitch upgrading.
> > > > > >> > > > >> > > >>
> > > > > >> > > > >> > > >> I left a comment in the doc, but we might want
> to
> > > > > iterate
> > > > > >> on
> > > > > >> > > the
> > > > > >> > > > >> > > >> terminology we use for high priority or
> > > "synchronous"
> > > > > DAG
> > > > > >> > runs
> > > > > >> > > to
> > > > > >> > > > >> > serve
> > > > > >> > > > >> > > LLM
> > > > > >> > > > >> > > >> responses - I find "interactive DAG runs" a bit
> > > > > confusing.
> > > > > >> > > > >> > > >>
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >
> > > > > >> > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > >> > > > >> > > To unsubscribe, e-mail:
> > > > dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org>
> > > > > >> <mailto:dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org>> <mailto:
> > > > > >> dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org> <mailto:
> > > > > >> dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org>>>
> > > > > >> > > > >> > > For additional commands, e-mail:
> > > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > > > >> <mailto:dev-help@airflow.apache.org <mailto:
> > > dev-help@airflow.apache.org>> <mailto:
> > > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > > > >> <mailto:dev-help@airflow.apache.org <mailto:
> > > dev-help@airflow.apache.org>>>
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Maciej Obuchowski <ob...@gmail.com>]

I would like to discuss the role of the Listener API and the OpenLineage
provider that utilizes it,
as there are current discussions that impact this integration.

For example, there is a consensus on (and I strongly support) removing
direct database access from
workers. However, currently, OpenLineage integration depends on it. I want
to explain what we do
and how we can proceed with removing database access from workers while not
only retaining but
possibly enhancing lineage functionality.

Currently, OpenLineage uses not only the data that the task directly has
but also utilizes
external connections in three ways:

- We send events to the configured OpenLineage backend. This does not
depend on any external
connection or Airflow database but merely on configuration, so it's
acceptable.
- We use the Airflow database, primarily to get connections - for example,
when gathering lineage
from SQLExecuteQueryOperator, to obtain information about the default
database or schema.
- We use external APIs - sometimes directly (like asking BigQuery about job
table dependencies) or
using Airflow Connections received from the Airflow database, like in
SQLExecuteQueryOperator to
get table schemas.

The issue arises if we are unable to perform the second or third methods -
not being able to
understand on which database a SELECT * from table operates would render
the potential lineage
data gathered from the statement itself meaningless.

As tasks require connection access, I assume connection data will somehow
be passed as part of the
metadata to task execution - whether it's part of the executor protocol or
in some other way (I'm
not an expert on that part of Airflow). Then, provided it's accessible as
part of some execution
context, and not only passed to the task's execute method, OpenLineage
could utilize it.

If the task execution metadata were standardized, we could take a step
further and perhaps move
from a model where TaskInstance listeners are executed on the worker to one
where they simply
depend on that metadata rather than the TaskInstance object. They could be
executed elsewhere - on
some specialized component working asynchronously, like the triggerer -
which would not be a fitting
name anymore.

pon., 13 maj 2024 o 08:55 Michał Modras <mi...@google.com.invalid>
napisał(a):

> Thanks to everyone for the constructive discussion - let's make it even
> more specific in the weekly syncs! I put a slot for us next Tuesday. I had
> some challenges with how some of the e-mails are displayed in this thread -
> please reach out to me if you'd like to be added.
>
> In the meantime, I summarised challenges, opportunities and potential
> work streams of Airflow 3 from the Composer team perspective. Please have a
> look and chime in here:
>
> https://docs.google.com/document/d/1kfqq3YFtUjFuuiv7KVZRqYgwFu9jcYyW_Mz7UUZYnks/edit?usp=sharing
>
> The main themes are on-demand & event driven DAG parsing, performance,
> isolation and stability improvements, and a big concern of backwards
> compatibility (+1 to Shubham's point - let's try to make changes in DAGS
> unnecessary, as opposed to potential changes in Airflow configuration).
>
> Best,
> Michal
>
> On Sat, May 11, 2024 at 9:30 AM Mehta, Shubham <sh...@amazon.com.invalid>
> wrote:
>
> > Sorry for joining the discussion late - I was down under the weather for
> > the past two weeks.
> > First off, great discussion so far and appreciate the effort and thought
> > everyone has put into exploring Airflow 3.
> >
> > The list from Kaxil, et al looks solid and I'm really looking forward to
> > more extensive discussions on those and seeing them implemented in
> Airflow.
> > They will truly unlock some amazing use cases with Airflow.
> >
> > Regarding building for Airflow 2 vs Airflow 3, personally, I think we
> > should keep pushing critical features for Airflow 2 until the scope for
> > Airflow 3 is properly defined. For example, we shouldn't pause work on
> the
> > multi-team capabilities as there are quite a lot of enterprises rely on
> > Airflow today, and we have been saying that this feature is coming for
> > years now. Introducing this in Airflow 3 means users have to jump through
> > another non-backward compatible release to take advantage of it, which
> > doesn't feel user focused. Airflow 3 is still at least a year away from
> > anything substantial shipping. Not every change has to be breaking - we
> can
> > develop some things for Airflow 2 and potentially adopt them as-is in
> > Airflow 3 if it makes sense. Only if we're certain the dev effort will be
> > way less by starting fresh on Airflow 3, should we consider that path.
> >
> > On Jarek's point about being very opinionated in Airflow 3.0 and
> > potentially dropping certain features/options to start with a lean core
> - I
> > am very divided on this. In most cases, I would say that being
> opinionated
> > is great, but in this case, we risk alienating our users, especially
> those
> > who self-host as they have proportionately less representation among the
> > maintainers. Thousands of organizations rely on Airflow and breaking the
> > trust could be costly in the long run. We should of course be basing
> those
> > decisions on real user data versus assumptions, but not everyone is going
> > to jump to 2.10+ in a few months and data will be biased. I am not
> > concerned about MWAA customers here, but more about users who are
> > self-hosting.
> >
> > We've put together a Google doc (with writing access for all) capturing
> > some wish list items based on customer discussions:
> >
> https://docs.google.com/document/d/17KKjwmwHquL_laMpsqZaZD5wzruKAAyFeOsLhGNp7vc/edit?usp=sharing
> > . It's still a work-in-progress as we gather more feedback from our
> > customers on what could help increase Airflow adoption in their orgs. We
> > understand that not everything in this list will be implemented, but it
> is
> > essential to bring forward all ideas and discuss to make Airflow better
> for
> > everyone.
> >
> > I think it would be great to collaborate with others including Jarek,
> > Kaxil, Jens, Bolke, Amogh etc. who have already or would like to share
> > ideas, to brainstorm tenets, major themes, scope of Airflow 3, along with
> > prioritized list. Maybe we can have a dedicated sync, say next Thursday
> > 8:30am PT, to discuss this further? I can volunteer to organize these
> > meetings. Initial set of meetings will be to brainstorm to bring forward
> a
> > comprehensive and aligned proposal forward for the community.
> >
> > I'm excited about Airflow 3 and looking forward to shaping Airflow's
> > future together!
> >
> > Shubham
> >
> >
> > On 2024-05-10, 8:45 AM, "Pierre Jeambrun" <pierrejbrun@gmail.com
> <mailto:
> > pierrejbrun@gmail.com>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > Thank you for the work and proposal made around airflow 3.
> >
> >
> > I also strongly believe that those important changes need to happen for
> > airflow to stay relevant and it would be the right moment to start
> > discussing and planning for airflow 3. As mentioned some features are
> hard
> > to develop within airflow 2 with backward compatibility in mind:
> > * I would love to see a full react front end, without FAB.
> > * We could also take the opportunity to rework and clarify our RBAC code,
> > we had quite a number of CVEs related to it.
> > * Do the breaking change for api clients and (finally) upgrade the
> openapi
> > client generator to a more recent version
> > * Extract business service layers that could be shared across the entire
> > codebase instead of having a 'component wise' approach. Such that
> handling
> > of dag dates (execution, logical), serialization, resource access control
> > and such tools could be implemented once and shared everywhere. (easier
> > maintenance, more consistency between API, CLI, and other core components
> > behaviors). Inspired by some famous software architecture such as
> hexagonal
> > or clean. Even if I don't think that those would be relevant to airflow
> > codebase stricto sensu.
> >
> >
> > As mentioned stability for our users is key and supporting airflow 2.x
> for
> > a while after airflow 3 is a must have. I just want to bring to
> > your attention that this means more work for our release managers and
> also
> > for the security team (backporting security patches, CVEs handling for
> two
> > different major versions, etc.). All that to say that we should choose
> this
> > period wisely. (long enough, but not too long)
> >
> >
> > This sounds like a really exciting time for airflow.
> > Cheers
> >
> >
> > Le jeu. 9 mai 2024 à 11:23, Amogh Desai <amoghdesai.oss@gmail.com
> <mailto:
> > amoghdesai.oss@gmail.com>> a écrit :
> >
> >
> > > Thanks for initiating this discussion @Kaxil Naik <kaxil@astronomer.io
> > <ma...@astronomer.io>>!
> > >
> > >
> > > Read through the document and the emails here and I like the direction
> in
> > > which
> > > we are proceeding.
> > >
> > > I also agree that removing code is almost as important as adding code.
> We
> > > need to have a clear checklist of what we want to
> > > remove and get a vote for the same from the community.
> > >
> > > I also see that we have multiple ways of doing a thing, which is not
> > needed
> > > (serialising for example) and removing
> > > which can lead to simpler interfaces for users, and contributors too.
> > >
> > > I also find the *secret masker* overly complex personally, why can't it
> > be
> > > simpler :)
> > >
> > > However, I have a small concern with the proportion of the users that
> > would
> > > like to migrate to an early version of
> > > Airflow 3 if we do not support various deployment modes. If we dont get
> > > enough initial feedback, we do not know how the new
> > > major is doing, no?
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <ash@apache.org
> > <ma...@apache.org>> wrote:
> > >
> > > > Strong agree - I think one of the things I regret most about Airflow
> 2
> > > was
> > > > that we didn't remove enough. It probably eased the upgrade process
> for
> > > > users though.
> > > >
> > > > > but I was hoping one day when we remove providers
> > > > > from main development, I can do it personally and beat Ash to it.
> > > >
> > > > Grrr ;l
> > > >
> > > > The other thing I want to look at (but don't have concrete ideas of
> > yet)
> > > > is some of the lesser used dag/task concurrency and dependency
> > features -
> > > > if we can remove a few things and get a faster scheduler than it's
> > worth
> > > > having a discussion over.
> > > >
> > > > On 9 May 2024 06:31:38 BST, Jarek Potiuk <jarek@potiuk.com <mailto:
> > jarek@potiuk.com>> wrote:
> > > > >I hope more voices will be coming soon as well :).
> > > > >
> > > > >Constance - few words of explanation, for my "remove" things. I do
> not
> > > say
> > > > >we "have to" remove those things or even that we "should".
> > > > >
> > > > >My main motivation for mentioning the list is that we often forget
> > that
> > > > >when we have a new major version of software - removing stuff is as
> > > > >important as adding. So I think when it comes to final decisions
> about
> > > > >Airflow 3 we should be well aware about both: what we add and what
> we
> > > > >remove. So my main point is - "removal" of things should be as
> > > important a
> > > > >part in our future discussions as "adding", and I'd say with 10
> years
> > of
> > > > >Airflow history, it's more important to remove things than to add
> > them.
> > > > >
> > > > >Side comment: Ash for one mentioned at multiple occasions how proud
> he
> > > is
> > > > >that his contributions to Airflow are net-negative - he removed more
> > > code
> > > > >than added. Unfortunately it's not shown any more here
> > > > >https://github.com/apache/airflow/graphs/contributors <
> > https://github.com/apache/airflow/graphs/contributors> - because we have
> > > > >more than 10.000 commits, but I was hoping one day when we remove
> > > > providers
> > > > >from main development, I can do it personally and beat Ash to it.
> > > > >
> > > > >Just wanted to stress a few things on why I think "discussing
> > potential
> > > > >removals" is important:
> > > > >
> > > > >* I think it is super important to deliver Airflow 3 really fast. It
> > > took
> > > > >
> > > > >1.5 year to release Airflow 2 and it was far too long and keeping
> > > Airflow
> > > > 1
> > > > >and 2 in parallel was a huge pain and drain and it slowed us down
> > > > >enormously. We should absolutely limit the time when we actively
> > develop
> > > > >both Airflow 2 features and Airflow 3 features.
> > > > >* I think most of the current deprecations of the 100+ we have does
> > not
> > > > >slow us down AT ALL. Removal of those is mostly cosmetic change, a
> > > little
> > > > >bit clutter to remove but they have little-to-no impact on actual
> > speed
> > > of
> > > > >development, it's just an old code that keeps on being around
> > > > >* on the other hand - some of the things I mentioned ARE slowing us
> > > down A
> > > > >LOT and will continue to do so until we remove them. For example, I
> > > > believe
> > > > >"postgresql + mysql + sqlite complete versioning solution for all
> the
> > > > >possible variants of storage" will be quite a bit more complex and
> > > testing
> > > > >will take far more time than if we drop mysql and choose a single
> > > storage
> > > > >approach for Airflow 3.0. I have no hard data to back it up, but my
> > gut
> > > > >feeling is that it can take at least twice as long to get it out in
> > the
> > > > >hands of our users if we try to have Airflow 2 parity in Airflow 3.0
> > for
> > > > >all the options we have there.
> > > > >* the telemetry will be cool - but even if we add it for 2.10, the
> > first
> > > > >time meaningful data will be available is maybe 2 years from now
> when
> > we
> > > > do
> > > > >Airflow 4. For now we can completely forget about it because it's
> only
> > > > >going to show us some early adopters of Airflow 2.10. But we have
> > > surveys
> > > > +
> > > > >we can collect data from Astronomer, Google. Amazon for some usage
> and
> > > > >identify who will be early adopters of Airflow 3 and target them
> > first.
> > > > >* the fact that we will drop something for Airflow 3.0, does not
> mean
> > > that
> > > > >we have to drop it "forever". We can safely assume that Airflow 3.0
> > will
> > > > be
> > > > >only used by early adopters, and many of our users will wait for
> 3.1,
> > > 3.2
> > > > >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those
> > > "slower
> > > > >moving users" from the 3.0. We can re-add things in 3.1 building on
> > > > >foundations of 3.0 after it is in the hands of those early
> adopters. I
> > > > >personally think we should prioritize speed of delivery of 3.0 over
> > > > >"complete support of every deployment option of Airflow 2" - to
> allow
> > > > >**some** of our early adopter users to migrate quickly, and add
> what's
> > > > >missing for those who would like to move later in later versions.
> > > > >* I think - this is the most important "product management" decision
> > > here
> > > > -
> > > > >based on the data we have available. Which users do we target for
> > > Airflow
> > > > >3.0, and what we can drop to deliver it faster (and possibly re-add
> > more
> > > > >stuff for 3.1+). I think it's crucial to the success of the Airflow
> > 3.0
> > > > >initiative and the most important decision from the Product
> management
> > > > side
> > > > >that will impact the timeline of Airflow 3.0. While a lot depends on
> > how
> > > > >much time and effort individuals will spend on implementing things,
> > the
> > > > >decision of what we drop will impact everyone's speed and overall
> > > delivery
> > > > >of Airflow 3.0. It's a decision we should not take lightly.
> > > > >
> > > > >So my main point is: let's add "what we remove" as a very important
> > > point
> > > > >in the product discussions we are going to have.
> > > > >
> > > > >J.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar
> > > > ><rbishund@amazon.com.inva <ma...@amazon.com.inva>lid>
> > wrote:
> > > > >
> > > > >> Like Constance and many others, there was time needed to process
> > this
> > > > >> fantastical summary and approach to Airflow 3. __
> > > > >> A lot of the points raised by Jarek make sense, ie being
> > prescriptive
> > > on
> > > > >> what features we want to be there on Day 1 vs launched on 3.x -
> this
> > > > does
> > > > >> allow us to move quickly for our users.
> > > > >> In line with Constance's concern, we need to be mindful of what we
> > > > decide
> > > > >> to keep and what we are willing to cut - which could be a
> > painstaking
> > > > >> process that could offset any gains of trying to be faster. I also
> > > > believe
> > > > >> that bringing all these new amazing features on Airflow 3 will
> peak
> > > the
> > > > >> interest of early adopters and eventually get others interested in
> > > > >> migration. However, I believe this migration will be a slow
> process
> > > and
> > > > >> will present a gap in certain functionalities that users may want
> > > before
> > > > >> entertaining any move to Airflow 3. There are still a lot of folks
> > > using
> > > > >> v1.10 today. There were several tactical initiatives in the past
> few
> > > > months
> > > > >> with intent on bringing new functionality, ie Multi-team, to
> Airflow
> > > 2.x
> > > > >> and I feel that while these efforts are not wasted, there should
> > still
> > > > be
> > > > >> an option to continue improving Airflow 2 to avoid alienating our
> > > users
> > > > on
> > > > >> the basis of a future promise in Airflow 3, that may not be easy
> to
> > > > migrate
> > > > >> towards.
> > > > >>
> > > > >> -- Rajesh
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > dev@airflow.apache.org <ma...@airflow.apache.org>> <
> > > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > > > >>
> > > > >>
> > > > >> On 2024-05-07, 12:26 PM, "Constance Martineau"
> > > > >> <constance@astronomer.io.INVA <mailto:
> constance@astronomer.io.INVA>
> > <mailto:constance@astronomer.io.INVA <mailto:
> constance@astronomer.io.INVA
> > >>
> > > > >> <mailto:constance@astronomer.io.INVA <mailto:
> > constance@astronomer.io.INVA> <mailto:
> > > > constance@astronomer.io.INVA <mailto:constance@astronomer.io.INVA
> > >>>LID>
> > > > >> wrote:
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> CAUTION: This email originated from outside of the organization.
> Do
> > > not
> > > > >> click links or open attachments unless you can confirm the sender
> > and
> > > > know
> > > > >> the content is safe.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > > > externe.
> > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne
> > > > pouvez
> > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > certain
> > > > que
> > > > >> le contenu ne présente aucun risque.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> Thank you Jarek for the detailed input. I've taken some time to
> > digest
> > > > your
> > > > >> points before responding.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> You've outlined a bold vision for Airflow 3, and I agree that
> being
> > > > >> decisive about the features and architectures will be the key to
> > > > success.
> > > > >> However, before we make final decisions on what features to cut or
> > > > retain,
> > > > >> it would be beneficial to have a more comprehensive understanding
> of
> > > how
> > > > >> the current features are utilized by the open-source community.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> @Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io>
> > <mailto:kaxil@astronomer.io <ma...@astronomer.io>>
> > > <mailto:
> > > > >> kaxil@astronomer.io <ma...@astronomer.io> <mailto:
> > kaxil@astronomer.io <ma...@astronomer.io>>>> recently
> > > initiated a
> > > > >> discussion on
> > > > >> collecting telemetry from open-source deployments:
> > > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m
> <
> > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> > > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>
> <
> > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;> <
> > > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m>
> <
> > https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;> <
> > > > >>
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt
> > <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&amp;gt
> >
> > > ;>.
> > > > >> This data
> > > > >> could be critical in ensuring our decisions are well-informed and
> > > > reflect
> > > > >> the real-world usage patterns of our users, not just those from
> > > managed
> > > > >> environments like Astro, MWAA or GCC.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> It's essential that we challenge our assumptions and base our
> > > decisions
> > > > on
> > > > >> a holistic view of feature usage. Identifying potential cuts is a
> > > > critical
> > > > >> step, but let's ensure our strategy aligns with the needs and
> > > > preferences
> > > > >> of the broader Airflow community.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <jarek@potiuk.com
> > <ma...@potiuk.com>
> > > <mailto:
> > > > >> jarek@potiuk.com <ma...@potiuk.com>> <mailto:
> > jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com
> > <ma...@potiuk.com>
> > > >>>
> > > > >> wrote:
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> > I am currently on sick leave, and still recovering - hoping to
> be
> > > > able to
> > > > >> > travel next week to the US as planned, so I just wanted to break
> > out
> > > > of
> > > > >> it
> > > > >> > to make one comment here.
> > > > >> >
> > > > >> > I got a clearer head now a bit with medications hopefully
> > working. I
> > > > am
> > > > >> > still taking it that should help me to get over the current
> state,
> > > > and I
> > > > >> > wanted to take a look at this discussion unraveling first. Over
> > last
> > > > week
> > > > >> > I disconnected from "day-to-day" Airflow and put some thoughts
> (as
> > > > much
> > > > >> as
> > > > >> > I could in my current state) on it. The whole subject of this
> > thread
> > > > was
> > > > >> > started from that - how the current discussions on AIP-67 and
> > others
> > > > >> change
> > > > >> > if we consider Airflow 3 is "starting".
> > > > >> >
> > > > >> > The price for back-compat is speed of development and quality.
> > More
> > > > >> > combinations to test, more unexpected issues uncovered,
> necessity
> > to
> > > > keep
> > > > >> > parallel paths (old/new) while adding new features. All what
> > > Constance
> > > > >> > wrote about and what Ash explained. We already started to trip
> > over
> > > > our
> > > > >> own
> > > > >> > feet mutliple times in a few last releases. Have we tested all
> > > > >> combinations
> > > > >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we
> > > already
> > > > see
> > > > >> > that in a number of "combos" of features things are not working
> in
> > > as
> > > > >> > stable a way as they did before.
> > > > >> >
> > > > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2
> > for a
> > > > long
> > > > >> > time (or even move out) as they will not want to move to Airflow
> > 3.
> > > A
> > > > lot
> > > > >> > of the work implemented in AIP-44 and design of AIP-67 was done
> > > around
> > > > >> > back-compatibility. but yes -
> > > > >> > it would have been way easier if designed anew without
> > > > back-compatibility
> > > > >> > in mind. And if we implement it and release it in Airflow 2 it
> > will
> > > > make
> > > > >> > new Airflow feature development even harder. That's why I wanted
> > to
> > > > treat
> > > > >> > it as "tactical" solution - hoping that in Airflow 3 we can make
> > it
> > > > >> > "properly" - and that's why I started the discussion here when I
> > > > sensed
> > > > >> > that we are "close" to Airflow 3 discussion, because I wanted to
> > see
> > > > what
> > > > >> > options we have there. This is why I have not yet concluded
> voting
> > > on
> > > > >> > AIP-67 waiting for the result of this discussion here.
> > > > >> >
> > > > >> > But if we are ready to go for Airflow.3 then I'd say there are
> two
> > > > >> > important things that should be part of the vision.
> > > > >> >
> > > > >> > 1) *We should be far more opinionated and have far fewer options
> > of
> > > > >> > running things in Airflow 3*. Even an order of magnitude more
> > > > >> opinionated.
> > > > >> > Make choices, stick to it, perfect those opinionated choices to
> > suit
> > > > >> 80/20
> > > > >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking
> not
> > > > fitting
> > > > >> > the 20% that might choose to stay at Airflow 2. We can choose
> now
> > > > which
> > > > >> > ~20% of cases we do not want to handle deliberately. And we
> should
> > > be
> > > > >> very,
> > > > >> > very strict about it. Default should be "no choice". This will
> > > > radically
> > > > >> > simplify deployment and should make it easier to simplify
> Airflow
> > > > >> > development and DAG authoring experience because we will have
> less
> > > > cases
> > > > >> to
> > > > >> > support. Even if we plan to add more options in the future, the
> > > first
> > > > >> > version of Airflow 3 should support one deployment approach
> only.
> > > > This is
> > > > >> > the only way we can deliver it fast. And we should be very bold
> > > there.
> > > > >> > Choose one option and go for it in pretty much every place we
> have
> > > > >> choices
> > > > >> > now. We should Aim for Airflow 3.0 to support only a subset of
> > > current
> > > > >> > users - but those who are most likely to migrate first and those
> > > with
> > > > the
> > > > >> > biggest need for the new features. We can think 3.x to support
> > more
> > > > >> cases,
> > > > >> > but 3.0 should be as opinionated as humanly possible.
> > > > >> >
> > > > >> > And this deployment option should be also something ALL our
> > > > stakeholders
> > > > >> > will feel OK with as a way forward in their offering.
> > > > >> >
> > > > >> > My candidates (and yes, some are bold):
> > > > >> >
> > > > >> > * *Drop MySQL*. If we have a single thing that makes us avoid
> our
> > > > schema
> > > > >> > and DB migration - this is the case. Let's choose Postgres 15+
> and
> > > use
> > > > >> some
> > > > >> > of the great features there. This will also enable much faster
> > async
> > > > SQL
> > > > >> > implementation and a number of other optimisations - not to
> > mention
> > > > >> cutting
> > > > >> > every single change in development and testing time by literally
> > > half.
> > > > >> And
> > > > >> > we should not look back to adding MySQL.
> > > > >> > * *Drop Celery/Sequential Executor* and start with Local + K8S
> > only
> > > > (and
> > > > >> > AWS/Google others can continue developing theirs of course in
> > > parallel
> > > > >> and
> > > > >> > continue Hybrid executor work). Later - we figure out a better
> > > > solution
> > > > >> to
> > > > >> > support "small" tasks using some new K8S features and possibly
> > > non-k8s
> > > > >> > solutions (Ray-based?)
> > > > >> > * *Cut Connection and Variable Management from DB/UI*. Leave
> only
> > > > Secrets
> > > > >> > Management. Later when we have a 100% extensible React UI, we
> can
> > > add
> > > > a
> > > > >> > "local DB secrets manager" add-on
> > > > >> > * *Choose a single way for DAG storage that will support
> > versioning
> > > > from
> > > > >> > day one*. Bear in mind we can add others later. Bolke's idea of
> > > using
> > > > >> > FSspec is an interesting one, we should see if it is feasible.
> > > > >> > * *Drop FAB completely (including custom plugins) and invest in
> > > > >> > implementing Auth Manager based on a dedicated, external
> solution*
> > > > >> > (KeyCloak
> > > > >> > that we've discussed before as a likely candidate)
> > > > >> > * *Leave Providers with Airflow 2 and add tests to make sure
> they
> > > are
> > > > >> > Airflow 3 future-compatible *- develop a way where we continue
> > > > >> development
> > > > >> > and contributions for Providers with Airflow 2 and add complete
> > > tests
> > > > to
> > > > >> > run them with Airflow 3. This way we can continue developing
> > > Provider
> > > > >> > features independently, and make them work for Airflow 2 (and
> > > continue
> > > > >> > adding features for Airflow 2 users alongside Airflow 2
> bugfixes),
> > > > while
> > > > >> > also gradually fix any Airflow3 incompatibilities and instead of
> > > > >> > "back-compatibility" tests make provider "forward-compatibility"
> > > > tests so
> > > > >> > that future Providers are tested and work on Airflow 3. Also it
> > will
> > > > make
> > > > >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested
> > > without
> > > > >> > investing in changing the current CI / test harness.
> > > > >> > * *Simplify Test Harness for Airflow 3 from the start *- without
> > > > >> providers
> > > > >> > and 790+ dependencies, we could vastly simplify Airflow3 testing
> > > > >> (basically
> > > > >> > make CI jobs from scratch) using mostly standard Python tooling
> > > > (while we
> > > > >> > can continue making use of the current test harness for Airflow
> 2
> > +
> > > > >> > Providers and extend it with Airflow 3 future-compatibility
> > tests).
> > > > That
> > > > >> > means Breeze would be only staying in Airflow 2 + Providers repo
> > as
> > > we
> > > > >> > should be able to achieve most of what we have there with local
> > > venv/
> > > > >> > tooling (especially with uv as underlying tooling).
> > > > >> >
> > > > >> > 2) *I think we only add very few new "important" features.
> > *Absolute
> > > > >> > minimum to make Airflow 3 appealing and add them only in Airflow
> > 3:
> > > > >> > versioning, multi-team, pluggable UI should only be Airflow 3 -
> it
> > > > makes
> > > > >> no
> > > > >> > sense to invest into Airflow 2 if we already know Airflow 3 is
> > > coming
> > > > -
> > > > >> > that generally triples effort needed to get them out. We should
> > drop
> > > > new
> > > > >> > features development in Airflow 2. This will give users
> incentive
> > to
> > > > move
> > > > >> > to 3 if the new features will be worth it. Even paying
> > > > >> > compatibility/migration price.
> > > > >> >
> > > > >> > Versionig, for example: I believe if we decide to go only with
> > > > Airflow 3
> > > > >> > and cut some of the above (Postgres only, Single versioning DAG
> > > > storage)
> > > > >> we
> > > > >> > can make bolder decisions in versioning and support simpler
> models
> > > > from
> > > > >> the
> > > > >> > get go (and deliver it faster). And we should add only a few -
> but
> > > > >> > important - features that our users clearly asked for and focus
> on
> > > > >> > delivering Airflow 3 as soon as possible (instead of Airflow
> 2.10
> > or
> > > > >> 2.11).
> > > > >> > Similarly - multi-team can be simplified if we cut things from
> the
> > > > list
> > > > >> > above and have Task isolation as first-class citizens in Airflow
> > > (and
> > > > the
> > > > >> > only option).
> > > > >> >
> > > > >> > My candidates very much concur with the list shared by Kaxil in
> > the
> > > > doc +
> > > > >> > I'd add multi-team (but simplified thanks to the cuts). But I
> also
> > > > here
> > > > >> > would mostly revert to Astronomer, Google. AWS team to define
> > > > >> collectively
> > > > >> > what is the absolute minimum set of features that would get the
> > > > "target"
> > > > >> > part of their customers happy. And ONLY do that.
> > > > >> >
> > > > >> > So in short - I think the big part of our discussion should be
> > what
> > > we
> > > > >> are
> > > > >> > ready to drop when we start airflow 3 and be very bold. Once we
> > know
> > > > we
> > > > >> > should figure out the absolute minimum of things that we can add
> > > that
> > > > >> will
> > > > >> > benefit a significant part of our users (and make use of
> increased
> > > > speed
> > > > >> > because we dropped things).
> > > > >> >
> > > > >> > J.
> > > > >> >
> > > > >> >
> > > > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau
> > > > >> > <constance@astronomer.io.inva <mailto:
> > constance@astronomer.io.inva> <mailto:constance@astronomer.io.inva
> > <ma...@astronomer.io.inva>>
> > > > >> <mailto:constance@astronomer.io.inva <mailto:
> > constance@astronomer.io.inva> <mailto:
> > > > constance@astronomer.io.inva <mailto:constance@astronomer.io.inva
> > >>>lid>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi Michal,
> > > > >> > >
> > > > >> > > Thanks for your thoughts on the Airflow 3 proposal. I
> appreciate
> > > > your
> > > > >> > > concerns about the migration overhead for our users with a
> major
> > > new
> > > > >> > > version and see the appeal in your suggestion to integrate
> many
> > of
> > > > the
> > > > >> > > proposed changes into Airflow 2 through separate AIPs. It’s a
> > > valid
> > > > >> point
> > > > >> > > and certainly aligns with the value of making incremental
> > > > improvements.
> > > > >> > >
> > > > >> > > However, after looking closely at the enhancements outlined
> for
> > > > Airflow
> > > > >> > 3,
> > > > >> > > I'm convinced they warrant a new major release. Here’s why:
> > > > >> > >
> > > > >> > > 1. *Core Architectural Changes:* We’re looking at foundational
> > > > changes
> > > > >> > > with Airflow 3—like redefining task priorities, separating
> task
> > > > >> > > definition
> > > > >> > > and task execution, and new AIPs like DAG versioning. remote
> > > > execution
> > > > >> > > and restricting database access from workers. These aren’t
> just
> > > > >> > > incremental
> > > > >> > > improvements but major shifts that will set the stage for the
> > next
> > > > >> > > decade
> > > > >> > > of Airflow’s architecture. Grouping these changes into a major
> > > > release
> > > > >> > > will
> > > > >> > > help us make these transitions more cleanly and with fewer
> > > > constraints
> > > > >> > > from
> > > > >> > > past decisions.
> > > > >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140
> > > > >> > deprecated
> > > > >> > > issues, and this will only grow if we continue without a major
> > > > >> > cleanup.
> > > > >> > > This makes it increasingly difficult to implement new features
> > > > >> > > effectively
> > > > >> > > while maintaining backward compatibility. A major release
> allows
> > > us
> > > > to
> > > > >> > > address these issues head-on, reducing technical debt and
> paving
> > > the
> > > > >> > way
> > > > >> > > for a more robust platform.
> > > > >> > > 3. *Managing Breaking Changes:* Let’s take the example of
> > > > restricting
> > > > >> > > database access from workers. It’s a necessary move for better
> > > > >> > security
> > > > >> > > and
> > > > >> > > also potentially scalability reasons (reduces DB load). Many
> > users
> > > > >> > have
> > > > >> > > workflows that interact with the DB, either by using raw sql
> or
> > by
> > > > >> > > leveraging a session object. We could implement this feature
> in
> > > > >> > Airflow
> > > > >> > > 2
> > > > >> > > and avoid breaking existing workflows by continuing to have
> the
> > > old
> > > > >> > > standard mode as default - much of the work is already done -
> > but
> > > > that
> > > > >> > > would mean supporting both the new secure mode and the old
> > > standard
> > > > >> > mode
> > > > >> > > indefinitely and design new features with the assumption that
> > most
> > > > >> > will
> > > > >> > > continue using the old standard mode. With Airflow 3, we can
> > make
> > > > >> > secure
> > > > >> > > mode the default or even the only option, simplifying
> > > implementation
> > > > >> > and
> > > > >> > > future development. This is just one example where it is
> > feasible
> > > to
> > > > >> > > implement in Airflow 2, but is better if we release it under
> the
> > > > >> > > context of
> > > > >> > > Airflow 3.
> > > > >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up
> > > > >> > > possibilities for handling workflows beyond batch processing.
> > > > Features
> > > > >> > > like
> > > > >> > > real-time DAG execution through API and multi-language task
> > > support
> > > > >> > are
> > > > >> > > big
> > > > >> > > steps forward, significantly expanding Airflow’s utility.
> > > > >> > >
> > > > >> > >
> > > > >> > > While integrating these updates into Airflow 2 might look less
> > > > >> disruptive
> > > > >> > > initially, the scale and nature of the required changes really
> > > > support
> > > > >> a
> > > > >> > > move to Airflow 3. It’s not just about adding new features;
> it’s
> > > > about
> > > > >> > > setting up Airflow so that it continues to remain relevant for
> > the
> > > > next
> > > > >> > ten
> > > > >> > > years.
> > > > >> > >
> > > > >> > > Constance
> > > > >> > >
> > > > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <
> > ash@apache.org <ma...@apache.org>
> > > > >> <mailto:ash@apache.org <ma...@apache.org>> <mailto:
> > ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <mailto:
> > ash@apache.org>
> > > > >>>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > There's a lot of technical debt hiding in Airflow,
> especially
> > > the
> > > > >> > > > scheduler that makes it harder and harder to efficiently add
> > new
> > > > >> > > features.
> > > > >> > > >
> > > > >> > > > At some point, very soon, we are going to have to remove
> some
> > > very
> > > > >> > > > infrequently used back compat shims that negatively affect
> > > > >> performance.
> > > > >> > > > Without doing that the pace at which we can realistically
> add
> > > > some of
> > > > >> > the
> > > > >> > > > more exciting features tends towards zero. Developer speed
> of
> > > > >> > > contributors
> > > > >> > > > is a factor here too!
> > > > >> > > >
> > > > >> > > > So while we are still using SemVer, that necessitates v3.
> > > > >> > > >
> > > > >> > > > Ash
> > > > >> > > >
> > > > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <
> > > > michalmodras@google.com <ma...@google.com>
> > > > >> <mailto:michalmodras@google.com <ma...@google.com>>
> > <mailto:michalmodras@google.com <ma...@google.com>
> > > > <mailto:
> > > > >> michalmodras@google.com <ma...@google.com>>>
> > > > >> > > .INVALID>
> > > > >> > > > wrote:
> > > > >> > > > >+1 to Jens's & Bolke's points here and in the doc
> > > > >> > > > >
> > > > >> > > > >I agree we should work on clarifying the directions we
> would
> > > like
> > > > >> > > Airflow
> > > > >> > > > >to go. Introducing a new major Airflow version is a massive
> > > > overhead
> > > > >> > for
> > > > >> > > > >users, who would need to plan for migrations, onboarding
> the
> > > new
> > > > >> > Airflow
> > > > >> > > > >(with a slightly different architecture), etc., and
> > effectively
> > > > >> > Airflow
> > > > >> > > 2
> > > > >> > > > >would live in parallel for a long time.
> > > > >> > > > >
> > > > >> > > > >Personally, I think most of the points in Kaxil's/Vikram's
> > doc
> > > > are
> > > > >> > > > valuable
> > > > >> > > > >projects of their own, and I could imagine all of them
> being
> > > > >> delivered
> > > > >> > > as
> > > > >> > > > >separate AIPs within Airflow 2 (surely new minor versions
> of
> > > > Airflow
> > > > >> > > 2). I
> > > > >> > > > >am not sure if the scope of changes and the goal we want to
> > > > achieve
> > > > >> is
> > > > >> > > a)
> > > > >> > > > >clear enough b) broad enough to call for a new major
> version.
> > > > >> > > > >
> > > > >> > > > >Best,
> > > > >> > > > >Michal
> > > > >> > > > >
> > > > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens
> > > (XC-AS/EAE-ADA-T)
> > > > >> > > > ><Jens.Scheffler@de.bosch.com.inva <mailto:
> > Jens.Scheffler@de.bosch.com.inva> <mailto:
> > > > >> Jens.Scheffler@de.bosch.com.inva <mailto:
> > Jens.Scheffler@de.bosch.com.inva>> <mailto:
> > > > Jens.Scheffler@de.bosch.com.inva <mailto:
> > Jens.Scheffler@de.bosch.com.inva>
> > > > >> <mailto:Jens.Scheffler@de.bosch.com.inva <mailto:
> > Jens.Scheffler@de.bosch.com.inva>>>lid> wrote:
> > > > >> > > > >
> > > > >> > > > >> Thanks for the document write-up, Kaxil. I assume this is
> > > > mostly a
> > > > >> > > > vision
> > > > >> > > > >> statement.
> > > > >> > > > >>
> > > > >> > > > >> Looking forward for a larger addendum where we can
> collect
> > > > things
> > > > >> > that
> > > > >> > > > we
> > > > >> > > > >> all can vote and agree on as targets.
> > > > >> > > > >>
> > > > >> > > > >> As I started earlier with a confluence page and it seems
> > this
> > > > is
> > > > >> not
> > > > >> > > > >> accessible to all, shall we convert this to a Google Doc
> > for
> > > > >> better
> > > > >> > > > >> collaboration and item collection?
> > > > >> > > > >>
> > > > >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> <
> > https://aka.ms/o0ukef&gt;> <
> > > > >> https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&amp;gt;&gt;> <
> > https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&amp;gt;&gt;> <
> > > > >> https://aka.ms/o0ukef&amp;gt;&gt;> <
> > https://aka.ms/o0ukef&amp;amp;gt;&amp;gt;&gt;>
> > > > >> > > > >> ________________________________
> > > > >> > > > >> From: Vikram Koka <vikram@astronomer.io.INVA <mailto:
> > vikram@astronomer.io.INVA> <mailto:
> > > > >> vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>>
> > <mailto:vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>
> > <mailto:
> > > > >> vikram@astronomer.io.INVA <mailto:vikram@astronomer.io.INVA
> >>>LID>
> > > > >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> > > > >> > > > >> To: dev@airflow.apache.org <mailto:
> dev@airflow.apache.org>
> > <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>
> > > > >> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>
> > <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>> <
> > > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > dev@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > dev@airflow.apache.org <ma...@airflow.apache.org>>>>
> > > > >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical
> > (Airflow
> > > > 2)
> > > > >> vs
> > > > >> > > > >> strategic (Airflow 3) approach
> > > > >> > > > >>
> > > > >> > > > >> Thank you for your feedback, Bolke and Andrey!
> > > > >> > > > >>
> > > > >> > > > >> Bolke,
> > > > >> > > > >> I have replied to some of your comments in the doc.
> > > > >> > > > >> I will provide a detailed write up on the "Interactive
> DAG
> > > run"
> > > > >> (or
> > > > >> > > > >> synchronous DAG run) capability, which has generated some
> > > early
> > > > >> > > > questions.
> > > > >> > > > >> I had intended to get an AIP published for that as a
> > > follow-up,
> > > > >> but
> > > > >> > I
> > > > >> > > > >> believe that a simpler write up would be useful ahead of
> > the
> > > > AIP.
> > > > >> > > > >>
> > > > >> > > > >> Andrey,
> > > > >> > > > >> You raise an interesting point.
> > > > >> > > > >>
> > > > >> > > > >> As part of the Airflow 2.0 release, we as a community had
> > > > decided
> > > > >> to
> > > > >> > > > >> strictly adhere to Semver as detailed in the document you
> > > > >> > referenced.
> > > > >> > > We
> > > > >> > > > >> also consciously split out the "Core Airflow" releases
> from
> > > the
> > > > >> > > > "Provider"
> > > > >> > > > >> releases at that time. We had a clear expectation then
> for
> > > the
> > > > >> > cadence
> > > > >> > > > of
> > > > >> > > > >> both minor and patch releases, which we have generally
> > > adhered
> > > > to
> > > > >> > > since
> > > > >> > > > >> then.
> > > > >> > > > >>
> > > > >> > > > >> Personally, I am more concerned about our Provider
> releases
> > > > right
> > > > >> > now,
> > > > >> > > > as
> > > > >> > > > >> compared to the cadence of our major releases. I believe
> > that
> > > > one
> > > > >> of
> > > > >> > > the
> > > > >> > > > >> proposed changes in the Airflow 3 document i.e. the clear
> > > > >> separation
> > > > >> > > for
> > > > >> > > > >> Task Execution will help here, but more may be needed.
> > > > >> > > > >>
> > > > >> > > > >> Definitely interested in more feedback on this as well.
> > > > >> > > > >>
> > > > >> > > > >> Vikram
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> > > > >> > > andrey.anshin@taragol.is <ma...@taragol.is>
> > <mailto:andrey.anshin@taragol.is <ma...@taragol.is>>
> > > <mailto:
> > > > >> andrey.anshin@taragol.is <ma...@taragol.is>
> <mailto:
> > andrey.anshin@taragol.is <ma...@taragol.is>>>
> > > > >> > > > >
> > > > >> > > > >> wrote:
> > > > >> > > > >>
> > > > >> > > > >> > I would like to propose to change (at least discuss)
> > > release
> > > > >> > policy
> > > > >> > > > >> around
> > > > >> > > > >> > the Major version of Airflow.
> > > > >> > > > >> >
> > > > >> > > > >> > Right now it is described as "These releases do not
> > happen
> > > > with
> > > > >> > any
> > > > >> > > > >> regular
> > > > >> > > > >> > interval or on any predictable schedule." :
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> > <
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> > >
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> > <
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0
> > >
> > > > >
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> > <
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0
> > >
> > > > >
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt
> > <
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;amp;reserved=0&amp;gt
> > >
> > > > >> ;>
> > > > >> > > > >> <
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > >
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > >
> > > > >
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > >
> > > > >
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&amp;gt
> > >
> > > > >> ;>
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >> > So maybe it is time to make it schedulable, e.g. one
> per
> > > two
> > > > >> years
> > > > >> > > or
> > > > >> > > > so.
> > > > >> > > > >> > This one could help us to avoid such a discussion in
> the
> > > > future,
> > > > >> > > like
> > > > >> > > > "We
> > > > >> > > > >> > don't know when Airflow 4 is coming.". At the moment
> when
> > > the
> > > > >> new
> > > > >> > > > major
> > > > >> > > > >> > version will be released new features wouldn't be added
> > in
> > > > the
> > > > >> old
> > > > >> > > > major
> > > > >> > > > >> > version, however we would support bug / security for a
> > > while,
> > > > >> > e.g. 1
> > > > >> > > > year
> > > > >> > > > >> > for bug fixes, 3 years for security fixes with a total
> 5
> > > year
> > > > >> > > > lifecycle
> > > > >> > > > >> per
> > > > >> > > > >> > a major version. These just are approximate time
> periods
> > > for
> > > > a
> > > > >> > > > definition
> > > > >> > > > >> > of current period, bugfix period and security fix
> period.
> > > > >> > > > >> >
> > > > >> > > > >> > In contributors' perspective it helps with dropping the
> > > > >> deprecated
> > > > >> > > > stuff
> > > > >> > > > >> > which resolves some old problem: we have to support
> > > > everything
> > > > >> > > > including
> > > > >> > > > >> > deprecated stuff and without schedulable lifecycle for
> > the
> > > > >> > > deprecated
> > > > >> > > > >> stuff
> > > > >> > > > >> > it could be showstopper for the new feature, because
> > > > sometimes
> > > > >> it
> > > > >> > > > hard to
> > > > >> > > > >> > support two different approaches for long period of
> time
> > > > with no
> > > > >> > > hope
> > > > >> > > > >> that
> > > > >> > > > >> > it will happen soon. For some fundamental stuff which
> do
> > > not
> > > > >> > > require a
> > > > >> > > > >> lot
> > > > >> > > > >> > things time to support we could postponed removal for
> > next
> > > > after
> > > > >> > the
> > > > >> > > > next
> > > > >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in
> > > > Airflow 5
> > > > >> > > > >> >
> > > > >> > > > >> > In the user perspective, they have at least bug fix
> > support
> > > > for
> > > > >> a
> > > > >> > > > while,
> > > > >> > > > >> if
> > > > >> > > > >> > someone want to use legacy version it their choice,
> > however
> > > > no
> > > > >> new
> > > > >> > > > >> > features, no new version of providers (after one year)
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >> > ----
> > > > >> > > > >> > Best Wishes
> > > > >> > > > >> > *Andrey Anshin*
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <
> > > > bdbruin@gmail.com <ma...@gmail.com>
> > > > >> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> > bdbruin@gmail.com <ma...@gmail.com> <mailto:
> > > > >> bdbruin@gmail.com <ma...@gmail.com>>>>
> > > > >> > > > wrote:
> > > > >> > > > >> >
> > > > >> > > > >> > > I have left several comments :-). And on interactive
> > dag
> > > > runs
> > > > >> > even
> > > > >> > > > >> after
> > > > >> > > > >> > > the explanation of Vikram I still don't have a clue
> > what
> > > we
> > > > >> want
> > > > >> > > to
> > > > >> > > > >> > > accomplish there :-P.
> > > > >> > > > >> > >
> > > > >> > > > >> > > I would like to see a mantra or team for Airflow 3.
> > That
> > > > helps
> > > > >> > > > nudging
> > > > >> > > > >> > > people in the same direction. Suggestions in the
> > > comments.
> > > > >> > > > >> > >
> > > > >> > > > >> > > Bolke
> > > > >> > > > >> > > Sent from my iPhone
> > > > >> > > > >> > >
> > > > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> > > > >> > > <vikram@astronomer.io.inva <ma...@astronomer.io.inva>
> > <mailto:vikram@astronomer.io.inva <ma...@astronomer.io.inva>>
> > > > <mailto:
> > > > >> vikram@astronomer.io.inva <ma...@astronomer.io.inva>
> > <mailto:vikram@astronomer.io.inva <mailto:vikram@astronomer.io.inva
> >>>lid
> > > > >> > > > >
> > > > >> > > > >> > > wrote:
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > Good point Jed.
> > > > >> > > > >> > > > I responded back to your comment in the doc as well
> > and
> > > > very
> > > > >> > > open
> > > > >> > > > to
> > > > >> > > > >> > > > changing the term in the doc.
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > Used the term "interactive DAG run" as the ability
> to
> > > > invoke
> > > > >> > or
> > > > >> > > > >> > trigger a
> > > > >> > > > >> > > > DAG run through the API, with the expectation of
> > > getting
> > > > >> back
> > > > >> > a
> > > > >> > > > >> result
> > > > >> > > > >> > > > immediately. An alternate term could be a
> > "synchronous
> > > > DAG
> > > > >> > run".
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > Regardless, this is a significant change so a good
> > term
> > > > to
> > > > >> > > > indicate
> > > > >> > > > >> the
> > > > >> > > > >> > > > expansion from "batch runs only" is warranted. Very
> > > open
> > > > to
> > > > >> > > > different
> > > > >> > > > >> > > terms
> > > > >> > > > >> > > > here.
> > > > >> > > > >> > > >
> > > > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > > > >> > > > >> > jedcunningham@apache.org <mailto:
> > jedcunningham@apache.org> <mailto:jedcunningham@apache.org <mailto:
> > jedcunningham@apache.org>>
> > > > >> <mailto:jedcunningham@apache.org <mailto:jedcunningham@apache.org
> >
> > <mailto:jedcunningham@apache.org <ma...@apache.org>>>
> > > > >> > > > >> > > >
> > > > >> > > > >> > > >> wrote:
> > > > >> > > > >> > > >>
> > > > >> > > > >> > > >> Very exciting! Looks like we will have a busy
> period
> > > of
> > > > >> time
> > > > >> > > > ahead
> > > > >> > > > >> of
> > > > >> > > > >> > > us.
> > > > >> > > > >> > > >> Overall I like the plan so far, especially using
> > this
> > > > >> year's
> > > > >> > > > Airflow
> > > > >> > > > >> > > Summit
> > > > >> > > > >> > > >> as an opportunity to announce and gather feedback,
> > and
> > > > the
> > > > >> > 2025
> > > > >> > > > >> > version
> > > > >> > > > >> > > to
> > > > >> > > > >> > > >> pitch upgrading.
> > > > >> > > > >> > > >>
> > > > >> > > > >> > > >> I left a comment in the doc, but we might want to
> > > > iterate
> > > > >> on
> > > > >> > > the
> > > > >> > > > >> > > >> terminology we use for high priority or
> > "synchronous"
> > > > DAG
> > > > >> > runs
> > > > >> > > to
> > > > >> > > > >> > serve
> > > > >> > > > >> > > LLM
> > > > >> > > > >> > > >> responses - I find "interactive DAG runs" a bit
> > > > confusing.
> > > > >> > > > >> > > >>
> > > > >> > > > >> > >
> > > > >> > > > >> > >
> > > > >> > > >
> > > > ---------------------------------------------------------------------
> > > > >> > > > >> > > To unsubscribe, e-mail:
> > > dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>
> > > > >> <mailto:dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>> <mailto:
> > > > >> dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org> <mailto:
> > > > >> dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>>>
> > > > >> > > > >> > > For additional commands, e-mail:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > > >> <mailto:dev-help@airflow.apache.org <mailto:
> > dev-help@airflow.apache.org>> <mailto:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > > >> <mailto:dev-help@airflow.apache.org <mailto:
> > dev-help@airflow.apache.org>>>
> > > > >> > > > >> > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> >
> >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Michał Modras <mi...@google.com.INVALID>]

Thanks to everyone for the constructive discussion - let's make it even
more specific in the weekly syncs! I put a slot for us next Tuesday. I had
some challenges with how some of the e-mails are displayed in this thread -
please reach out to me if you'd like to be added.

In the meantime, I summarised challenges, opportunities and potential
work streams of Airflow 3 from the Composer team perspective. Please have a
look and chime in here:
https://docs.google.com/document/d/1kfqq3YFtUjFuuiv7KVZRqYgwFu9jcYyW_Mz7UUZYnks/edit?usp=sharing

The main themes are on-demand & event driven DAG parsing, performance,
isolation and stability improvements, and a big concern of backwards
compatibility (+1 to Shubham's point - let's try to make changes in DAGS
unnecessary, as opposed to potential changes in Airflow configuration).

Best,
Michal

On Sat, May 11, 2024 at 9:30 AM Mehta, Shubham <sh...@amazon.com.invalid>
wrote:

> Sorry for joining the discussion late - I was down under the weather for
> the past two weeks.
> First off, great discussion so far and appreciate the effort and thought
> everyone has put into exploring Airflow 3.
>
> The list from Kaxil, et al looks solid and I'm really looking forward to
> more extensive discussions on those and seeing them implemented in Airflow.
> They will truly unlock some amazing use cases with Airflow.
>
> Regarding building for Airflow 2 vs Airflow 3, personally, I think we
> should keep pushing critical features for Airflow 2 until the scope for
> Airflow 3 is properly defined. For example, we shouldn't pause work on the
> multi-team capabilities as there are quite a lot of enterprises rely on
> Airflow today, and we have been saying that this feature is coming for
> years now. Introducing this in Airflow 3 means users have to jump through
> another non-backward compatible release to take advantage of it, which
> doesn't feel user focused. Airflow 3 is still at least a year away from
> anything substantial shipping. Not every change has to be breaking - we can
> develop some things for Airflow 2 and potentially adopt them as-is in
> Airflow 3 if it makes sense. Only if we're certain the dev effort will be
> way less by starting fresh on Airflow 3, should we consider that path.
>
> On Jarek's point about being very opinionated in Airflow 3.0 and
> potentially dropping certain features/options to start with a lean core - I
> am very divided on this. In most cases, I would say that being opinionated
> is great, but in this case, we risk alienating our users, especially those
> who self-host as they have proportionately less representation among the
> maintainers. Thousands of organizations rely on Airflow and breaking the
> trust could be costly in the long run. We should of course be basing those
> decisions on real user data versus assumptions, but not everyone is going
> to jump to 2.10+ in a few months and data will be biased. I am not
> concerned about MWAA customers here, but more about users who are
> self-hosting.
>
> We've put together a Google doc (with writing access for all) capturing
> some wish list items based on customer discussions:
> https://docs.google.com/document/d/17KKjwmwHquL_laMpsqZaZD5wzruKAAyFeOsLhGNp7vc/edit?usp=sharing
> . It's still a work-in-progress as we gather more feedback from our
> customers on what could help increase Airflow adoption in their orgs. We
> understand that not everything in this list will be implemented, but it is
> essential to bring forward all ideas and discuss to make Airflow better for
> everyone.
>
> I think it would be great to collaborate with others including Jarek,
> Kaxil, Jens, Bolke, Amogh etc. who have already or would like to share
> ideas, to brainstorm tenets, major themes, scope of Airflow 3, along with
> prioritized list. Maybe we can have a dedicated sync, say next Thursday
> 8:30am PT, to discuss this further? I can volunteer to organize these
> meetings. Initial set of meetings will be to brainstorm to bring forward a
> comprehensive and aligned proposal forward for the community.
>
> I'm excited about Airflow 3 and looking forward to shaping Airflow's
> future together!
>
> Shubham
>
>
> On 2024-05-10, 8:45 AM, "Pierre Jeambrun" <pierrejbrun@gmail.com <mailto:
> pierrejbrun@gmail.com>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
> Thank you for the work and proposal made around airflow 3.
>
>
> I also strongly believe that those important changes need to happen for
> airflow to stay relevant and it would be the right moment to start
> discussing and planning for airflow 3. As mentioned some features are hard
> to develop within airflow 2 with backward compatibility in mind:
> * I would love to see a full react front end, without FAB.
> * We could also take the opportunity to rework and clarify our RBAC code,
> we had quite a number of CVEs related to it.
> * Do the breaking change for api clients and (finally) upgrade the openapi
> client generator to a more recent version
> * Extract business service layers that could be shared across the entire
> codebase instead of having a 'component wise' approach. Such that handling
> of dag dates (execution, logical), serialization, resource access control
> and such tools could be implemented once and shared everywhere. (easier
> maintenance, more consistency between API, CLI, and other core components
> behaviors). Inspired by some famous software architecture such as hexagonal
> or clean. Even if I don't think that those would be relevant to airflow
> codebase stricto sensu.
>
>
> As mentioned stability for our users is key and supporting airflow 2.x for
> a while after airflow 3 is a must have. I just want to bring to
> your attention that this means more work for our release managers and also
> for the security team (backporting security patches, CVEs handling for two
> different major versions, etc.). All that to say that we should choose this
> period wisely. (long enough, but not too long)
>
>
> This sounds like a really exciting time for airflow.
> Cheers
>
>
> Le jeu. 9 mai 2024 à 11:23, Amogh Desai <amoghdesai.oss@gmail.com <mailto:
> amoghdesai.oss@gmail.com>> a écrit :
>
>
> > Thanks for initiating this discussion @Kaxil Naik <kaxil@astronomer.io
> <ma...@astronomer.io>>!
> >
> >
> > Read through the document and the emails here and I like the direction in
> > which
> > we are proceeding.
> >
> > I also agree that removing code is almost as important as adding code. We
> > need to have a clear checklist of what we want to
> > remove and get a vote for the same from the community.
> >
> > I also see that we have multiple ways of doing a thing, which is not
> needed
> > (serialising for example) and removing
> > which can lead to simpler interfaces for users, and contributors too.
> >
> > I also find the *secret masker* overly complex personally, why can't it
> be
> > simpler :)
> >
> > However, I have a small concern with the proportion of the users that
> would
> > like to migrate to an early version of
> > Airflow 3 if we do not support various deployment modes. If we dont get
> > enough initial feedback, we do not know how the new
> > major is doing, no?
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <ash@apache.org
> <ma...@apache.org>> wrote:
> >
> > > Strong agree - I think one of the things I regret most about Airflow 2
> > was
> > > that we didn't remove enough. It probably eased the upgrade process for
> > > users though.
> > >
> > > > but I was hoping one day when we remove providers
> > > > from main development, I can do it personally and beat Ash to it.
> > >
> > > Grrr ;l
> > >
> > > The other thing I want to look at (but don't have concrete ideas of
> yet)
> > > is some of the lesser used dag/task concurrency and dependency
> features -
> > > if we can remove a few things and get a faster scheduler than it's
> worth
> > > having a discussion over.
> > >
> > > On 9 May 2024 06:31:38 BST, Jarek Potiuk <jarek@potiuk.com <mailto:
> jarek@potiuk.com>> wrote:
> > > >I hope more voices will be coming soon as well :).
> > > >
> > > >Constance - few words of explanation, for my "remove" things. I do not
> > say
> > > >we "have to" remove those things or even that we "should".
> > > >
> > > >My main motivation for mentioning the list is that we often forget
> that
> > > >when we have a new major version of software - removing stuff is as
> > > >important as adding. So I think when it comes to final decisions about
> > > >Airflow 3 we should be well aware about both: what we add and what we
> > > >remove. So my main point is - "removal" of things should be as
> > important a
> > > >part in our future discussions as "adding", and I'd say with 10 years
> of
> > > >Airflow history, it's more important to remove things than to add
> them.
> > > >
> > > >Side comment: Ash for one mentioned at multiple occasions how proud he
> > is
> > > >that his contributions to Airflow are net-negative - he removed more
> > code
> > > >than added. Unfortunately it's not shown any more here
> > > >https://github.com/apache/airflow/graphs/contributors <
> https://github.com/apache/airflow/graphs/contributors> - because we have
> > > >more than 10.000 commits, but I was hoping one day when we remove
> > > providers
> > > >from main development, I can do it personally and beat Ash to it.
> > > >
> > > >Just wanted to stress a few things on why I think "discussing
> potential
> > > >removals" is important:
> > > >
> > > >* I think it is super important to deliver Airflow 3 really fast. It
> > took
> > > >
> > > >1.5 year to release Airflow 2 and it was far too long and keeping
> > Airflow
> > > 1
> > > >and 2 in parallel was a huge pain and drain and it slowed us down
> > > >enormously. We should absolutely limit the time when we actively
> develop
> > > >both Airflow 2 features and Airflow 3 features.
> > > >* I think most of the current deprecations of the 100+ we have does
> not
> > > >slow us down AT ALL. Removal of those is mostly cosmetic change, a
> > little
> > > >bit clutter to remove but they have little-to-no impact on actual
> speed
> > of
> > > >development, it's just an old code that keeps on being around
> > > >* on the other hand - some of the things I mentioned ARE slowing us
> > down A
> > > >LOT and will continue to do so until we remove them. For example, I
> > > believe
> > > >"postgresql + mysql + sqlite complete versioning solution for all the
> > > >possible variants of storage" will be quite a bit more complex and
> > testing
> > > >will take far more time than if we drop mysql and choose a single
> > storage
> > > >approach for Airflow 3.0. I have no hard data to back it up, but my
> gut
> > > >feeling is that it can take at least twice as long to get it out in
> the
> > > >hands of our users if we try to have Airflow 2 parity in Airflow 3.0
> for
> > > >all the options we have there.
> > > >* the telemetry will be cool - but even if we add it for 2.10, the
> first
> > > >time meaningful data will be available is maybe 2 years from now when
> we
> > > do
> > > >Airflow 4. For now we can completely forget about it because it's only
> > > >going to show us some early adopters of Airflow 2.10. But we have
> > surveys
> > > +
> > > >we can collect data from Astronomer, Google. Amazon for some usage and
> > > >identify who will be early adopters of Airflow 3 and target them
> first.
> > > >* the fact that we will drop something for Airflow 3.0, does not mean
> > that
> > > >we have to drop it "forever". We can safely assume that Airflow 3.0
> will
> > > be
> > > >only used by early adopters, and many of our users will wait for 3.1,
> > 3.2
> > > >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those
> > "slower
> > > >moving users" from the 3.0. We can re-add things in 3.1 building on
> > > >foundations of 3.0 after it is in the hands of those early adopters. I
> > > >personally think we should prioritize speed of delivery of 3.0 over
> > > >"complete support of every deployment option of Airflow 2" - to allow
> > > >**some** of our early adopter users to migrate quickly, and add what's
> > > >missing for those who would like to move later in later versions.
> > > >* I think - this is the most important "product management" decision
> > here
> > > -
> > > >based on the data we have available. Which users do we target for
> > Airflow
> > > >3.0, and what we can drop to deliver it faster (and possibly re-add
> more
> > > >stuff for 3.1+). I think it's crucial to the success of the Airflow
> 3.0
> > > >initiative and the most important decision from the Product management
> > > side
> > > >that will impact the timeline of Airflow 3.0. While a lot depends on
> how
> > > >much time and effort individuals will spend on implementing things,
> the
> > > >decision of what we drop will impact everyone's speed and overall
> > delivery
> > > >of Airflow 3.0. It's a decision we should not take lightly.
> > > >
> > > >So my main point is: let's add "what we remove" as a very important
> > point
> > > >in the product discussions we are going to have.
> > > >
> > > >J.
> > > >
> > > >
> > > >
> > > >
> > > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar
> > > ><rbishund@amazon.com.inva <ma...@amazon.com.inva>lid>
> wrote:
> > > >
> > > >> Like Constance and many others, there was time needed to process
> this
> > > >> fantastical summary and approach to Airflow 3. __
> > > >> A lot of the points raised by Jarek make sense, ie being
> prescriptive
> > on
> > > >> what features we want to be there on Day 1 vs launched on 3.x - this
> > > does
> > > >> allow us to move quickly for our users.
> > > >> In line with Constance's concern, we need to be mindful of what we
> > > decide
> > > >> to keep and what we are willing to cut - which could be a
> painstaking
> > > >> process that could offset any gains of trying to be faster. I also
> > > believe
> > > >> that bringing all these new amazing features on Airflow 3 will peak
> > the
> > > >> interest of early adopters and eventually get others interested in
> > > >> migration. However, I believe this migration will be a slow process
> > and
> > > >> will present a gap in certain functionalities that users may want
> > before
> > > >> entertaining any move to Airflow 3. There are still a lot of folks
> > using
> > > >> v1.10 today. There were several tactical initiatives in the past few
> > > months
> > > >> with intent on bringing new functionality, ie Multi-team, to Airflow
> > 2.x
> > > >> and I feel that while these efforts are not wasted, there should
> still
> > > be
> > > >> an option to continue improving Airflow 2 to avoid alienating our
> > users
> > > on
> > > >> the basis of a future promise in Airflow 3, that may not be easy to
> > > migrate
> > > >> towards.
> > > >>
> > > >> -- Rajesh
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> dev@airflow.apache.org <ma...@airflow.apache.org>> <
> > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > > >>
> > > >>
> > > >> On 2024-05-07, 12:26 PM, "Constance Martineau"
> > > >> <constance@astronomer.io.INVA <ma...@astronomer.io.INVA>
> <mailto:constance@astronomer.io.INVA <mailto:constance@astronomer.io.INVA
> >>
> > > >> <mailto:constance@astronomer.io.INVA <mailto:
> constance@astronomer.io.INVA> <mailto:
> > > constance@astronomer.io.INVA <mailto:constance@astronomer.io.INVA
> >>>LID>
> > > >> wrote:
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> CAUTION: This email originated from outside of the organization. Do
> > not
> > > >> click links or open attachments unless you can confirm the sender
> and
> > > know
> > > >> the content is safe.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > > externe.
> > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> > > pouvez
> > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain
> > > que
> > > >> le contenu ne présente aucun risque.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> Thank you Jarek for the detailed input. I've taken some time to
> digest
> > > your
> > > >> points before responding.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> You've outlined a bold vision for Airflow 3, and I agree that being
> > > >> decisive about the features and architectures will be the key to
> > > success.
> > > >> However, before we make final decisions on what features to cut or
> > > retain,
> > > >> it would be beneficial to have a more comprehensive understanding of
> > how
> > > >> the current features are utilized by the open-source community.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> @Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io>
> <mailto:kaxil@astronomer.io <ma...@astronomer.io>>
> > <mailto:
> > > >> kaxil@astronomer.io <ma...@astronomer.io> <mailto:
> kaxil@astronomer.io <ma...@astronomer.io>>>> recently
> > initiated a
> > > >> discussion on
> > > >> collecting telemetry from open-source deployments:
> > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m <
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;> <
> > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;> <
> > > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt
> <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&amp;gt>
> > ;>.
> > > >> This data
> > > >> could be critical in ensuring our decisions are well-informed and
> > > reflect
> > > >> the real-world usage patterns of our users, not just those from
> > managed
> > > >> environments like Astro, MWAA or GCC.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> It's essential that we challenge our assumptions and base our
> > decisions
> > > on
> > > >> a holistic view of feature usage. Identifying potential cuts is a
> > > critical
> > > >> step, but let's ensure our strategy aligns with the needs and
> > > preferences
> > > >> of the broader Airflow community.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>
> > <mailto:
> > > >> jarek@potiuk.com <ma...@potiuk.com>> <mailto:
> jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com
> <ma...@potiuk.com>
> > >>>
> > > >> wrote:
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> > I am currently on sick leave, and still recovering - hoping to be
> > > able to
> > > >> > travel next week to the US as planned, so I just wanted to break
> out
> > > of
> > > >> it
> > > >> > to make one comment here.
> > > >> >
> > > >> > I got a clearer head now a bit with medications hopefully
> working. I
> > > am
> > > >> > still taking it that should help me to get over the current state,
> > > and I
> > > >> > wanted to take a look at this discussion unraveling first. Over
> last
> > > week
> > > >> > I disconnected from "day-to-day" Airflow and put some thoughts (as
> > > much
> > > >> as
> > > >> > I could in my current state) on it. The whole subject of this
> thread
> > > was
> > > >> > started from that - how the current discussions on AIP-67 and
> others
> > > >> change
> > > >> > if we consider Airflow 3 is "starting".
> > > >> >
> > > >> > The price for back-compat is speed of development and quality.
> More
> > > >> > combinations to test, more unexpected issues uncovered, necessity
> to
> > > keep
> > > >> > parallel paths (old/new) while adding new features. All what
> > Constance
> > > >> > wrote about and what Ash explained. We already started to trip
> over
> > > our
> > > >> own
> > > >> > feet mutliple times in a few last releases. Have we tested all
> > > >> combinations
> > > >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we
> > already
> > > see
> > > >> > that in a number of "combos" of features things are not working in
> > as
> > > >> > stable a way as they did before.
> > > >> >
> > > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2
> for a
> > > long
> > > >> > time (or even move out) as they will not want to move to Airflow
> 3.
> > A
> > > lot
> > > >> > of the work implemented in AIP-44 and design of AIP-67 was done
> > around
> > > >> > back-compatibility. but yes -
> > > >> > it would have been way easier if designed anew without
> > > back-compatibility
> > > >> > in mind. And if we implement it and release it in Airflow 2 it
> will
> > > make
> > > >> > new Airflow feature development even harder. That's why I wanted
> to
> > > treat
> > > >> > it as "tactical" solution - hoping that in Airflow 3 we can make
> it
> > > >> > "properly" - and that's why I started the discussion here when I
> > > sensed
> > > >> > that we are "close" to Airflow 3 discussion, because I wanted to
> see
> > > what
> > > >> > options we have there. This is why I have not yet concluded voting
> > on
> > > >> > AIP-67 waiting for the result of this discussion here.
> > > >> >
> > > >> > But if we are ready to go for Airflow.3 then I'd say there are two
> > > >> > important things that should be part of the vision.
> > > >> >
> > > >> > 1) *We should be far more opinionated and have far fewer options
> of
> > > >> > running things in Airflow 3*. Even an order of magnitude more
> > > >> opinionated.
> > > >> > Make choices, stick to it, perfect those opinionated choices to
> suit
> > > >> 80/20
> > > >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not
> > > fitting
> > > >> > the 20% that might choose to stay at Airflow 2. We can choose now
> > > which
> > > >> > ~20% of cases we do not want to handle deliberately. And we should
> > be
> > > >> very,
> > > >> > very strict about it. Default should be "no choice". This will
> > > radically
> > > >> > simplify deployment and should make it easier to simplify Airflow
> > > >> > development and DAG authoring experience because we will have less
> > > cases
> > > >> to
> > > >> > support. Even if we plan to add more options in the future, the
> > first
> > > >> > version of Airflow 3 should support one deployment approach only.
> > > This is
> > > >> > the only way we can deliver it fast. And we should be very bold
> > there.
> > > >> > Choose one option and go for it in pretty much every place we have
> > > >> choices
> > > >> > now. We should Aim for Airflow 3.0 to support only a subset of
> > current
> > > >> > users - but those who are most likely to migrate first and those
> > with
> > > the
> > > >> > biggest need for the new features. We can think 3.x to support
> more
> > > >> cases,
> > > >> > but 3.0 should be as opinionated as humanly possible.
> > > >> >
> > > >> > And this deployment option should be also something ALL our
> > > stakeholders
> > > >> > will feel OK with as a way forward in their offering.
> > > >> >
> > > >> > My candidates (and yes, some are bold):
> > > >> >
> > > >> > * *Drop MySQL*. If we have a single thing that makes us avoid our
> > > schema
> > > >> > and DB migration - this is the case. Let's choose Postgres 15+ and
> > use
> > > >> some
> > > >> > of the great features there. This will also enable much faster
> async
> > > SQL
> > > >> > implementation and a number of other optimisations - not to
> mention
> > > >> cutting
> > > >> > every single change in development and testing time by literally
> > half.
> > > >> And
> > > >> > we should not look back to adding MySQL.
> > > >> > * *Drop Celery/Sequential Executor* and start with Local + K8S
> only
> > > (and
> > > >> > AWS/Google others can continue developing theirs of course in
> > parallel
> > > >> and
> > > >> > continue Hybrid executor work). Later - we figure out a better
> > > solution
> > > >> to
> > > >> > support "small" tasks using some new K8S features and possibly
> > non-k8s
> > > >> > solutions (Ray-based?)
> > > >> > * *Cut Connection and Variable Management from DB/UI*. Leave only
> > > Secrets
> > > >> > Management. Later when we have a 100% extensible React UI, we can
> > add
> > > a
> > > >> > "local DB secrets manager" add-on
> > > >> > * *Choose a single way for DAG storage that will support
> versioning
> > > from
> > > >> > day one*. Bear in mind we can add others later. Bolke's idea of
> > using
> > > >> > FSspec is an interesting one, we should see if it is feasible.
> > > >> > * *Drop FAB completely (including custom plugins) and invest in
> > > >> > implementing Auth Manager based on a dedicated, external solution*
> > > >> > (KeyCloak
> > > >> > that we've discussed before as a likely candidate)
> > > >> > * *Leave Providers with Airflow 2 and add tests to make sure they
> > are
> > > >> > Airflow 3 future-compatible *- develop a way where we continue
> > > >> development
> > > >> > and contributions for Providers with Airflow 2 and add complete
> > tests
> > > to
> > > >> > run them with Airflow 3. This way we can continue developing
> > Provider
> > > >> > features independently, and make them work for Airflow 2 (and
> > continue
> > > >> > adding features for Airflow 2 users alongside Airflow 2 bugfixes),
> > > while
> > > >> > also gradually fix any Airflow3 incompatibilities and instead of
> > > >> > "back-compatibility" tests make provider "forward-compatibility"
> > > tests so
> > > >> > that future Providers are tested and work on Airflow 3. Also it
> will
> > > make
> > > >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested
> > without
> > > >> > investing in changing the current CI / test harness.
> > > >> > * *Simplify Test Harness for Airflow 3 from the start *- without
> > > >> providers
> > > >> > and 790+ dependencies, we could vastly simplify Airflow3 testing
> > > >> (basically
> > > >> > make CI jobs from scratch) using mostly standard Python tooling
> > > (while we
> > > >> > can continue making use of the current test harness for Airflow 2
> +
> > > >> > Providers and extend it with Airflow 3 future-compatibility
> tests).
> > > That
> > > >> > means Breeze would be only staying in Airflow 2 + Providers repo
> as
> > we
> > > >> > should be able to achieve most of what we have there with local
> > venv/
> > > >> > tooling (especially with uv as underlying tooling).
> > > >> >
> > > >> > 2) *I think we only add very few new "important" features.
> *Absolute
> > > >> > minimum to make Airflow 3 appealing and add them only in Airflow
> 3:
> > > >> > versioning, multi-team, pluggable UI should only be Airflow 3 - it
> > > makes
> > > >> no
> > > >> > sense to invest into Airflow 2 if we already know Airflow 3 is
> > coming
> > > -
> > > >> > that generally triples effort needed to get them out. We should
> drop
> > > new
> > > >> > features development in Airflow 2. This will give users incentive
> to
> > > move
> > > >> > to 3 if the new features will be worth it. Even paying
> > > >> > compatibility/migration price.
> > > >> >
> > > >> > Versionig, for example: I believe if we decide to go only with
> > > Airflow 3
> > > >> > and cut some of the above (Postgres only, Single versioning DAG
> > > storage)
> > > >> we
> > > >> > can make bolder decisions in versioning and support simpler models
> > > from
> > > >> the
> > > >> > get go (and deliver it faster). And we should add only a few - but
> > > >> > important - features that our users clearly asked for and focus on
> > > >> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10
> or
> > > >> 2.11).
> > > >> > Similarly - multi-team can be simplified if we cut things from the
> > > list
> > > >> > above and have Task isolation as first-class citizens in Airflow
> > (and
> > > the
> > > >> > only option).
> > > >> >
> > > >> > My candidates very much concur with the list shared by Kaxil in
> the
> > > doc +
> > > >> > I'd add multi-team (but simplified thanks to the cuts). But I also
> > > here
> > > >> > would mostly revert to Astronomer, Google. AWS team to define
> > > >> collectively
> > > >> > what is the absolute minimum set of features that would get the
> > > "target"
> > > >> > part of their customers happy. And ONLY do that.
> > > >> >
> > > >> > So in short - I think the big part of our discussion should be
> what
> > we
> > > >> are
> > > >> > ready to drop when we start airflow 3 and be very bold. Once we
> know
> > > we
> > > >> > should figure out the absolute minimum of things that we can add
> > that
> > > >> will
> > > >> > benefit a significant part of our users (and make use of increased
> > > speed
> > > >> > because we dropped things).
> > > >> >
> > > >> > J.
> > > >> >
> > > >> >
> > > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau
> > > >> > <constance@astronomer.io.inva <mailto:
> constance@astronomer.io.inva> <mailto:constance@astronomer.io.inva
> <ma...@astronomer.io.inva>>
> > > >> <mailto:constance@astronomer.io.inva <mailto:
> constance@astronomer.io.inva> <mailto:
> > > constance@astronomer.io.inva <mailto:constance@astronomer.io.inva
> >>>lid>
> > > >> wrote:
> > > >> >
> > > >> > > Hi Michal,
> > > >> > >
> > > >> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate
> > > your
> > > >> > > concerns about the migration overhead for our users with a major
> > new
> > > >> > > version and see the appeal in your suggestion to integrate many
> of
> > > the
> > > >> > > proposed changes into Airflow 2 through separate AIPs. It’s a
> > valid
> > > >> point
> > > >> > > and certainly aligns with the value of making incremental
> > > improvements.
> > > >> > >
> > > >> > > However, after looking closely at the enhancements outlined for
> > > Airflow
> > > >> > 3,
> > > >> > > I'm convinced they warrant a new major release. Here’s why:
> > > >> > >
> > > >> > > 1. *Core Architectural Changes:* We’re looking at foundational
> > > changes
> > > >> > > with Airflow 3—like redefining task priorities, separating task
> > > >> > > definition
> > > >> > > and task execution, and new AIPs like DAG versioning. remote
> > > execution
> > > >> > > and restricting database access from workers. These aren’t just
> > > >> > > incremental
> > > >> > > improvements but major shifts that will set the stage for the
> next
> > > >> > > decade
> > > >> > > of Airflow’s architecture. Grouping these changes into a major
> > > release
> > > >> > > will
> > > >> > > help us make these transitions more cleanly and with fewer
> > > constraints
> > > >> > > from
> > > >> > > past decisions.
> > > >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140
> > > >> > deprecated
> > > >> > > issues, and this will only grow if we continue without a major
> > > >> > cleanup.
> > > >> > > This makes it increasingly difficult to implement new features
> > > >> > > effectively
> > > >> > > while maintaining backward compatibility. A major release allows
> > us
> > > to
> > > >> > > address these issues head-on, reducing technical debt and paving
> > the
> > > >> > way
> > > >> > > for a more robust platform.
> > > >> > > 3. *Managing Breaking Changes:* Let’s take the example of
> > > restricting
> > > >> > > database access from workers. It’s a necessary move for better
> > > >> > security
> > > >> > > and
> > > >> > > also potentially scalability reasons (reduces DB load). Many
> users
> > > >> > have
> > > >> > > workflows that interact with the DB, either by using raw sql or
> by
> > > >> > > leveraging a session object. We could implement this feature in
> > > >> > Airflow
> > > >> > > 2
> > > >> > > and avoid breaking existing workflows by continuing to have the
> > old
> > > >> > > standard mode as default - much of the work is already done -
> but
> > > that
> > > >> > > would mean supporting both the new secure mode and the old
> > standard
> > > >> > mode
> > > >> > > indefinitely and design new features with the assumption that
> most
> > > >> > will
> > > >> > > continue using the old standard mode. With Airflow 3, we can
> make
> > > >> > secure
> > > >> > > mode the default or even the only option, simplifying
> > implementation
> > > >> > and
> > > >> > > future development. This is just one example where it is
> feasible
> > to
> > > >> > > implement in Airflow 2, but is better if we release it under the
> > > >> > > context of
> > > >> > > Airflow 3.
> > > >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up
> > > >> > > possibilities for handling workflows beyond batch processing.
> > > Features
> > > >> > > like
> > > >> > > real-time DAG execution through API and multi-language task
> > support
> > > >> > are
> > > >> > > big
> > > >> > > steps forward, significantly expanding Airflow’s utility.
> > > >> > >
> > > >> > >
> > > >> > > While integrating these updates into Airflow 2 might look less
> > > >> disruptive
> > > >> > > initially, the scale and nature of the required changes really
> > > support
> > > >> a
> > > >> > > move to Airflow 3. It’s not just about adding new features; it’s
> > > about
> > > >> > > setting up Airflow so that it continues to remain relevant for
> the
> > > next
> > > >> > ten
> > > >> > > years.
> > > >> > >
> > > >> > > Constance
> > > >> > >
> > > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <
> ash@apache.org <ma...@apache.org>
> > > >> <mailto:ash@apache.org <ma...@apache.org>> <mailto:
> ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <mailto:
> ash@apache.org>
> > > >>>
> > > >> wrote:
> > > >> > >
> > > >> > > > There's a lot of technical debt hiding in Airflow, especially
> > the
> > > >> > > > scheduler that makes it harder and harder to efficiently add
> new
> > > >> > > features.
> > > >> > > >
> > > >> > > > At some point, very soon, we are going to have to remove some
> > very
> > > >> > > > infrequently used back compat shims that negatively affect
> > > >> performance.
> > > >> > > > Without doing that the pace at which we can realistically add
> > > some of
> > > >> > the
> > > >> > > > more exciting features tends towards zero. Developer speed of
> > > >> > > contributors
> > > >> > > > is a factor here too!
> > > >> > > >
> > > >> > > > So while we are still using SemVer, that necessitates v3.
> > > >> > > >
> > > >> > > > Ash
> > > >> > > >
> > > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <
> > > michalmodras@google.com <ma...@google.com>
> > > >> <mailto:michalmodras@google.com <ma...@google.com>>
> <mailto:michalmodras@google.com <ma...@google.com>
> > > <mailto:
> > > >> michalmodras@google.com <ma...@google.com>>>
> > > >> > > .INVALID>
> > > >> > > > wrote:
> > > >> > > > >+1 to Jens's & Bolke's points here and in the doc
> > > >> > > > >
> > > >> > > > >I agree we should work on clarifying the directions we would
> > like
> > > >> > > Airflow
> > > >> > > > >to go. Introducing a new major Airflow version is a massive
> > > overhead
> > > >> > for
> > > >> > > > >users, who would need to plan for migrations, onboarding the
> > new
> > > >> > Airflow
> > > >> > > > >(with a slightly different architecture), etc., and
> effectively
> > > >> > Airflow
> > > >> > > 2
> > > >> > > > >would live in parallel for a long time.
> > > >> > > > >
> > > >> > > > >Personally, I think most of the points in Kaxil's/Vikram's
> doc
> > > are
> > > >> > > > valuable
> > > >> > > > >projects of their own, and I could imagine all of them being
> > > >> delivered
> > > >> > > as
> > > >> > > > >separate AIPs within Airflow 2 (surely new minor versions of
> > > Airflow
> > > >> > > 2). I
> > > >> > > > >am not sure if the scope of changes and the goal we want to
> > > achieve
> > > >> is
> > > >> > > a)
> > > >> > > > >clear enough b) broad enough to call for a new major version.
> > > >> > > > >
> > > >> > > > >Best,
> > > >> > > > >Michal
> > > >> > > > >
> > > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens
> > (XC-AS/EAE-ADA-T)
> > > >> > > > ><Jens.Scheffler@de.bosch.com.inva <mailto:
> Jens.Scheffler@de.bosch.com.inva> <mailto:
> > > >> Jens.Scheffler@de.bosch.com.inva <mailto:
> Jens.Scheffler@de.bosch.com.inva>> <mailto:
> > > Jens.Scheffler@de.bosch.com.inva <mailto:
> Jens.Scheffler@de.bosch.com.inva>
> > > >> <mailto:Jens.Scheffler@de.bosch.com.inva <mailto:
> Jens.Scheffler@de.bosch.com.inva>>>lid> wrote:
> > > >> > > > >
> > > >> > > > >> Thanks for the document write-up, Kaxil. I assume this is
> > > mostly a
> > > >> > > > vision
> > > >> > > > >> statement.
> > > >> > > > >>
> > > >> > > > >> Looking forward for a larger addendum where we can collect
> > > things
> > > >> > that
> > > >> > > > we
> > > >> > > > >> all can vote and agree on as targets.
> > > >> > > > >>
> > > >> > > > >> As I started earlier with a confluence page and it seems
> this
> > > is
> > > >> not
> > > >> > > > >> accessible to all, shall we convert this to a Google Doc
> for
> > > >> better
> > > >> > > > >> collaboration and item collection?
> > > >> > > > >>
> > > >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> <
> https://aka.ms/o0ukef&gt;> <
> > > >> https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&amp;gt;&gt;> <
> https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&amp;gt;&gt;> <
> > > >> https://aka.ms/o0ukef&amp;gt;&gt;> <
> https://aka.ms/o0ukef&amp;amp;gt;&amp;gt;&gt;>
> > > >> > > > >> ________________________________
> > > >> > > > >> From: Vikram Koka <vikram@astronomer.io.INVA <mailto:
> vikram@astronomer.io.INVA> <mailto:
> > > >> vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>>
> <mailto:vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>
> <mailto:
> > > >> vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>>>LID>
> > > >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> > > >> > > > >> To: dev@airflow.apache.org <ma...@airflow.apache.org>
> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>
> > > >> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>
> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>> <
> > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> dev@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> dev@airflow.apache.org <ma...@airflow.apache.org>>>>
> > > >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical
> (Airflow
> > > 2)
> > > >> vs
> > > >> > > > >> strategic (Airflow 3) approach
> > > >> > > > >>
> > > >> > > > >> Thank you for your feedback, Bolke and Andrey!
> > > >> > > > >>
> > > >> > > > >> Bolke,
> > > >> > > > >> I have replied to some of your comments in the doc.
> > > >> > > > >> I will provide a detailed write up on the "Interactive DAG
> > run"
> > > >> (or
> > > >> > > > >> synchronous DAG run) capability, which has generated some
> > early
> > > >> > > > questions.
> > > >> > > > >> I had intended to get an AIP published for that as a
> > follow-up,
> > > >> but
> > > >> > I
> > > >> > > > >> believe that a simpler write up would be useful ahead of
> the
> > > AIP.
> > > >> > > > >>
> > > >> > > > >> Andrey,
> > > >> > > > >> You raise an interesting point.
> > > >> > > > >>
> > > >> > > > >> As part of the Airflow 2.0 release, we as a community had
> > > decided
> > > >> to
> > > >> > > > >> strictly adhere to Semver as detailed in the document you
> > > >> > referenced.
> > > >> > > We
> > > >> > > > >> also consciously split out the "Core Airflow" releases from
> > the
> > > >> > > > "Provider"
> > > >> > > > >> releases at that time. We had a clear expectation then for
> > the
> > > >> > cadence
> > > >> > > > of
> > > >> > > > >> both minor and patch releases, which we have generally
> > adhered
> > > to
> > > >> > > since
> > > >> > > > >> then.
> > > >> > > > >>
> > > >> > > > >> Personally, I am more concerned about our Provider releases
> > > right
> > > >> > now,
> > > >> > > > as
> > > >> > > > >> compared to the cadence of our major releases. I believe
> that
> > > one
> > > >> of
> > > >> > > the
> > > >> > > > >> proposed changes in the Airflow 3 document i.e. the clear
> > > >> separation
> > > >> > > for
> > > >> > > > >> Task Execution will help here, but more may be needed.
> > > >> > > > >>
> > > >> > > > >> Definitely interested in more feedback on this as well.
> > > >> > > > >>
> > > >> > > > >> Vikram
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> > > >> > > andrey.anshin@taragol.is <ma...@taragol.is>
> <mailto:andrey.anshin@taragol.is <ma...@taragol.is>>
> > <mailto:
> > > >> andrey.anshin@taragol.is <ma...@taragol.is> <mailto:
> andrey.anshin@taragol.is <ma...@taragol.is>>>
> > > >> > > > >
> > > >> > > > >> wrote:
> > > >> > > > >>
> > > >> > > > >> > I would like to propose to change (at least discuss)
> > release
> > > >> > policy
> > > >> > > > >> around
> > > >> > > > >> > the Major version of Airflow.
> > > >> > > > >> >
> > > >> > > > >> > Right now it is described as "These releases do not
> happen
> > > with
> > > >> > any
> > > >> > > > >> regular
> > > >> > > > >> > interval or on any predictable schedule." :
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> >
> > > >> <
> > > >>
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0
> >
> > > >
> > > >> <
> > > >>
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0
> >
> > > >
> > > >> <
> > > >>
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;amp;reserved=0&amp;gt
> >
> > > >> ;>
> > > >> > > > >> <
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> <
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> >
> > > >> <
> > > >>
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> <
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> >
> > > >
> > > >> <
> > > >>
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> <
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> >
> > > >
> > > >> <
> > > >>
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt
> <
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&amp;gt
> >
> > > >> ;>
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > So maybe it is time to make it schedulable, e.g. one per
> > two
> > > >> years
> > > >> > > or
> > > >> > > > so.
> > > >> > > > >> > This one could help us to avoid such a discussion in the
> > > future,
> > > >> > > like
> > > >> > > > "We
> > > >> > > > >> > don't know when Airflow 4 is coming.". At the moment when
> > the
> > > >> new
> > > >> > > > major
> > > >> > > > >> > version will be released new features wouldn't be added
> in
> > > the
> > > >> old
> > > >> > > > major
> > > >> > > > >> > version, however we would support bug / security for a
> > while,
> > > >> > e.g. 1
> > > >> > > > year
> > > >> > > > >> > for bug fixes, 3 years for security fixes with a total 5
> > year
> > > >> > > > lifecycle
> > > >> > > > >> per
> > > >> > > > >> > a major version. These just are approximate time periods
> > for
> > > a
> > > >> > > > definition
> > > >> > > > >> > of current period, bugfix period and security fix period.
> > > >> > > > >> >
> > > >> > > > >> > In contributors' perspective it helps with dropping the
> > > >> deprecated
> > > >> > > > stuff
> > > >> > > > >> > which resolves some old problem: we have to support
> > > everything
> > > >> > > > including
> > > >> > > > >> > deprecated stuff and without schedulable lifecycle for
> the
> > > >> > > deprecated
> > > >> > > > >> stuff
> > > >> > > > >> > it could be showstopper for the new feature, because
> > > sometimes
> > > >> it
> > > >> > > > hard to
> > > >> > > > >> > support two different approaches for long period of time
> > > with no
> > > >> > > hope
> > > >> > > > >> that
> > > >> > > > >> > it will happen soon. For some fundamental stuff which do
> > not
> > > >> > > require a
> > > >> > > > >> lot
> > > >> > > > >> > things time to support we could postponed removal for
> next
> > > after
> > > >> > the
> > > >> > > > next
> > > >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in
> > > Airflow 5
> > > >> > > > >> >
> > > >> > > > >> > In the user perspective, they have at least bug fix
> support
> > > for
> > > >> a
> > > >> > > > while,
> > > >> > > > >> if
> > > >> > > > >> > someone want to use legacy version it their choice,
> however
> > > no
> > > >> new
> > > >> > > > >> > features, no new version of providers (after one year)
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > ----
> > > >> > > > >> > Best Wishes
> > > >> > > > >> > *Andrey Anshin*
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <
> > > bdbruin@gmail.com <ma...@gmail.com>
> > > >> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:
> bdbruin@gmail.com <ma...@gmail.com> <mailto:
> > > >> bdbruin@gmail.com <ma...@gmail.com>>>>
> > > >> > > > wrote:
> > > >> > > > >> >
> > > >> > > > >> > > I have left several comments :-). And on interactive
> dag
> > > runs
> > > >> > even
> > > >> > > > >> after
> > > >> > > > >> > > the explanation of Vikram I still don't have a clue
> what
> > we
> > > >> want
> > > >> > > to
> > > >> > > > >> > > accomplish there :-P.
> > > >> > > > >> > >
> > > >> > > > >> > > I would like to see a mantra or team for Airflow 3.
> That
> > > helps
> > > >> > > > nudging
> > > >> > > > >> > > people in the same direction. Suggestions in the
> > comments.
> > > >> > > > >> > >
> > > >> > > > >> > > Bolke
> > > >> > > > >> > > Sent from my iPhone
> > > >> > > > >> > >
> > > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> > > >> > > <vikram@astronomer.io.inva <ma...@astronomer.io.inva>
> <mailto:vikram@astronomer.io.inva <ma...@astronomer.io.inva>>
> > > <mailto:
> > > >> vikram@astronomer.io.inva <ma...@astronomer.io.inva>
> <mailto:vikram@astronomer.io.inva <ma...@astronomer.io.inva>>>lid
> > > >> > > > >
> > > >> > > > >> > > wrote:
> > > >> > > > >> > > >
> > > >> > > > >> > > > Good point Jed.
> > > >> > > > >> > > > I responded back to your comment in the doc as well
> and
> > > very
> > > >> > > open
> > > >> > > > to
> > > >> > > > >> > > > changing the term in the doc.
> > > >> > > > >> > > >
> > > >> > > > >> > > > Used the term "interactive DAG run" as the ability to
> > > invoke
> > > >> > or
> > > >> > > > >> > trigger a
> > > >> > > > >> > > > DAG run through the API, with the expectation of
> > getting
> > > >> back
> > > >> > a
> > > >> > > > >> result
> > > >> > > > >> > > > immediately. An alternate term could be a
> "synchronous
> > > DAG
> > > >> > run".
> > > >> > > > >> > > >
> > > >> > > > >> > > > Regardless, this is a significant change so a good
> term
> > > to
> > > >> > > > indicate
> > > >> > > > >> the
> > > >> > > > >> > > > expansion from "batch runs only" is warranted. Very
> > open
> > > to
> > > >> > > > different
> > > >> > > > >> > > terms
> > > >> > > > >> > > > here.
> > > >> > > > >> > > >
> > > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > > >> > > > >> > jedcunningham@apache.org <mailto:
> jedcunningham@apache.org> <mailto:jedcunningham@apache.org <mailto:
> jedcunningham@apache.org>>
> > > >> <mailto:jedcunningham@apache.org <ma...@apache.org>
> <mailto:jedcunningham@apache.org <ma...@apache.org>>>
> > > >> > > > >> > > >
> > > >> > > > >> > > >> wrote:
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> Very exciting! Looks like we will have a busy period
> > of
> > > >> time
> > > >> > > > ahead
> > > >> > > > >> of
> > > >> > > > >> > > us.
> > > >> > > > >> > > >> Overall I like the plan so far, especially using
> this
> > > >> year's
> > > >> > > > Airflow
> > > >> > > > >> > > Summit
> > > >> > > > >> > > >> as an opportunity to announce and gather feedback,
> and
> > > the
> > > >> > 2025
> > > >> > > > >> > version
> > > >> > > > >> > > to
> > > >> > > > >> > > >> pitch upgrading.
> > > >> > > > >> > > >>
> > > >> > > > >> > > >> I left a comment in the doc, but we might want to
> > > iterate
> > > >> on
> > > >> > > the
> > > >> > > > >> > > >> terminology we use for high priority or
> "synchronous"
> > > DAG
> > > >> > runs
> > > >> > > to
> > > >> > > > >> > serve
> > > >> > > > >> > > LLM
> > > >> > > > >> > > >> responses - I find "interactive DAG runs" a bit
> > > confusing.
> > > >> > > > >> > > >>
> > > >> > > > >> > >
> > > >> > > > >> > >
> > > >> > > >
> > > ---------------------------------------------------------------------
> > > >> > > > >> > > To unsubscribe, e-mail:
> > dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>
> > > >> <mailto:dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>> <mailto:
> > > >> dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org> <mailto:
> > > >> dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>>>
> > > >> > > > >> > > For additional commands, e-mail:
> > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > >> <mailto:dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org>> <mailto:
> > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > >> <mailto:dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org>>>
> > > >> > > > >> > >
> > > >> > > > >> > >
> > > >> > > > >> >
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> >
>
>
>
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

["Mehta, Shubham" <sh...@amazon.com.INVALID>]

Sorry for joining the discussion late - I was down under the weather for the past two weeks.
First off, great discussion so far and appreciate the effort and thought everyone has put into exploring Airflow 3. 

The list from Kaxil, et al looks solid and I'm really looking forward to more extensive discussions on those and seeing them implemented in Airflow. They will truly unlock some amazing use cases with Airflow.

Regarding building for Airflow 2 vs Airflow 3, personally, I think we should keep pushing critical features for Airflow 2 until the scope for Airflow 3 is properly defined. For example, we shouldn't pause work on the multi-team capabilities as there are quite a lot of enterprises rely on Airflow today, and we have been saying that this feature is coming for years now. Introducing this in Airflow 3 means users have to jump through another non-backward compatible release to take advantage of it, which doesn't feel user focused. Airflow 3 is still at least a year away from anything substantial shipping. Not every change has to be breaking - we can develop some things for Airflow 2 and potentially adopt them as-is in Airflow 3 if it makes sense. Only if we're certain the dev effort will be way less by starting fresh on Airflow 3, should we consider that path.

On Jarek's point about being very opinionated in Airflow 3.0 and potentially dropping certain features/options to start with a lean core - I am very divided on this. In most cases, I would say that being opinionated is great, but in this case, we risk alienating our users, especially those who self-host as they have proportionately less representation among the maintainers. Thousands of organizations rely on Airflow and breaking the trust could be costly in the long run. We should of course be basing those decisions on real user data versus assumptions, but not everyone is going to jump to 2.10+ in a few months and data will be biased. I am not concerned about MWAA customers here, but more about users who are self-hosting.

We've put together a Google doc (with writing access for all) capturing some wish list items based on customer discussions: https://docs.google.com/document/d/17KKjwmwHquL_laMpsqZaZD5wzruKAAyFeOsLhGNp7vc/edit?usp=sharing . It's still a work-in-progress as we gather more feedback from our customers on what could help increase Airflow adoption in their orgs. We understand that not everything in this list will be implemented, but it is essential to bring forward all ideas and discuss to make Airflow better for everyone.

I think it would be great to collaborate with others including Jarek, Kaxil, Jens, Bolke, Amogh etc. who have already or would like to share ideas, to brainstorm tenets, major themes, scope of Airflow 3, along with prioritized list. Maybe we can have a dedicated sync, say next Thursday 8:30am PT, to discuss this further? I can volunteer to organize these meetings. Initial set of meetings will be to brainstorm to bring forward a comprehensive and aligned proposal forward for the community.

I'm excited about Airflow 3 and looking forward to shaping Airflow's future together!

Shubham


On 2024-05-10, 8:45 AM, "Pierre Jeambrun" <pierrejbrun@gmail.com <ma...@gmail.com>> wrote:


CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.






AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.






Thank you for the work and proposal made around airflow 3.


I also strongly believe that those important changes need to happen for
airflow to stay relevant and it would be the right moment to start
discussing and planning for airflow 3. As mentioned some features are hard
to develop within airflow 2 with backward compatibility in mind:
* I would love to see a full react front end, without FAB.
* We could also take the opportunity to rework and clarify our RBAC code,
we had quite a number of CVEs related to it.
* Do the breaking change for api clients and (finally) upgrade the openapi
client generator to a more recent version
* Extract business service layers that could be shared across the entire
codebase instead of having a 'component wise' approach. Such that handling
of dag dates (execution, logical), serialization, resource access control
and such tools could be implemented once and shared everywhere. (easier
maintenance, more consistency between API, CLI, and other core components
behaviors). Inspired by some famous software architecture such as hexagonal
or clean. Even if I don't think that those would be relevant to airflow
codebase stricto sensu.


As mentioned stability for our users is key and supporting airflow 2.x for
a while after airflow 3 is a must have. I just want to bring to
your attention that this means more work for our release managers and also
for the security team (backporting security patches, CVEs handling for two
different major versions, etc.). All that to say that we should choose this
period wisely. (long enough, but not too long)


This sounds like a really exciting time for airflow.
Cheers


Le jeu. 9 mai 2024 à 11:23, Amogh Desai <amoghdesai.oss@gmail.com <ma...@gmail.com>> a écrit :


> Thanks for initiating this discussion @Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io>>!
>
>
> Read through the document and the emails here and I like the direction in
> which
> we are proceeding.
>
> I also agree that removing code is almost as important as adding code. We
> need to have a clear checklist of what we want to
> remove and get a vote for the same from the community.
>
> I also see that we have multiple ways of doing a thing, which is not needed
> (serialising for example) and removing
> which can lead to simpler interfaces for users, and contributors too.
>
> I also find the *secret masker* overly complex personally, why can't it be
> simpler :)
>
> However, I have a small concern with the proportion of the users that would
> like to migrate to an early version of
> Airflow 3 if we do not support various deployment modes. If we dont get
> enough initial feedback, we do not know how the new
> major is doing, no?
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>> wrote:
>
> > Strong agree - I think one of the things I regret most about Airflow 2
> was
> > that we didn't remove enough. It probably eased the upgrade process for
> > users though.
> >
> > > but I was hoping one day when we remove providers
> > > from main development, I can do it personally and beat Ash to it.
> >
> > Grrr ;l
> >
> > The other thing I want to look at (but don't have concrete ideas of yet)
> > is some of the lesser used dag/task concurrency and dependency features -
> > if we can remove a few things and get a faster scheduler than it's worth
> > having a discussion over.
> >
> > On 9 May 2024 06:31:38 BST, Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
> > >I hope more voices will be coming soon as well :).
> > >
> > >Constance - few words of explanation, for my "remove" things. I do not
> say
> > >we "have to" remove those things or even that we "should".
> > >
> > >My main motivation for mentioning the list is that we often forget that
> > >when we have a new major version of software - removing stuff is as
> > >important as adding. So I think when it comes to final decisions about
> > >Airflow 3 we should be well aware about both: what we add and what we
> > >remove. So my main point is - "removal" of things should be as
> important a
> > >part in our future discussions as "adding", and I'd say with 10 years of
> > >Airflow history, it's more important to remove things than to add them.
> > >
> > >Side comment: Ash for one mentioned at multiple occasions how proud he
> is
> > >that his contributions to Airflow are net-negative - he removed more
> code
> > >than added. Unfortunately it's not shown any more here
> > >https://github.com/apache/airflow/graphs/contributors <https://github.com/apache/airflow/graphs/contributors> - because we have
> > >more than 10.000 commits, but I was hoping one day when we remove
> > providers
> > >from main development, I can do it personally and beat Ash to it.
> > >
> > >Just wanted to stress a few things on why I think "discussing potential
> > >removals" is important:
> > >
> > >* I think it is super important to deliver Airflow 3 really fast. It
> took
> > >
> > >1.5 year to release Airflow 2 and it was far too long and keeping
> Airflow
> > 1
> > >and 2 in parallel was a huge pain and drain and it slowed us down
> > >enormously. We should absolutely limit the time when we actively develop
> > >both Airflow 2 features and Airflow 3 features.
> > >* I think most of the current deprecations of the 100+ we have does not
> > >slow us down AT ALL. Removal of those is mostly cosmetic change, a
> little
> > >bit clutter to remove but they have little-to-no impact on actual speed
> of
> > >development, it's just an old code that keeps on being around
> > >* on the other hand - some of the things I mentioned ARE slowing us
> down A
> > >LOT and will continue to do so until we remove them. For example, I
> > believe
> > >"postgresql + mysql + sqlite complete versioning solution for all the
> > >possible variants of storage" will be quite a bit more complex and
> testing
> > >will take far more time than if we drop mysql and choose a single
> storage
> > >approach for Airflow 3.0. I have no hard data to back it up, but my gut
> > >feeling is that it can take at least twice as long to get it out in the
> > >hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for
> > >all the options we have there.
> > >* the telemetry will be cool - but even if we add it for 2.10, the first
> > >time meaningful data will be available is maybe 2 years from now when we
> > do
> > >Airflow 4. For now we can completely forget about it because it's only
> > >going to show us some early adopters of Airflow 2.10. But we have
> surveys
> > +
> > >we can collect data from Astronomer, Google. Amazon for some usage and
> > >identify who will be early adopters of Airflow 3 and target them first.
> > >* the fact that we will drop something for Airflow 3.0, does not mean
> that
> > >we have to drop it "forever". We can safely assume that Airflow 3.0 will
> > be
> > >only used by early adopters, and many of our users will wait for 3.1,
> 3.2
> > >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those
> "slower
> > >moving users" from the 3.0. We can re-add things in 3.1 building on
> > >foundations of 3.0 after it is in the hands of those early adopters. I
> > >personally think we should prioritize speed of delivery of 3.0 over
> > >"complete support of every deployment option of Airflow 2" - to allow
> > >**some** of our early adopter users to migrate quickly, and add what's
> > >missing for those who would like to move later in later versions.
> > >* I think - this is the most important "product management" decision
> here
> > -
> > >based on the data we have available. Which users do we target for
> Airflow
> > >3.0, and what we can drop to deliver it faster (and possibly re-add more
> > >stuff for 3.1+). I think it's crucial to the success of the Airflow 3.0
> > >initiative and the most important decision from the Product management
> > side
> > >that will impact the timeline of Airflow 3.0. While a lot depends on how
> > >much time and effort individuals will spend on implementing things, the
> > >decision of what we drop will impact everyone's speed and overall
> delivery
> > >of Airflow 3.0. It's a decision we should not take lightly.
> > >
> > >So my main point is: let's add "what we remove" as a very important
> point
> > >in the product discussions we are going to have.
> > >
> > >J.
> > >
> > >
> > >
> > >
> > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar
> > ><rbishund@amazon.com.inva <ma...@amazon.com.inva>lid> wrote:
> > >
> > >> Like Constance and many others, there was time needed to process this
> > >> fantastical summary and approach to Airflow 3. __
> > >> A lot of the points raised by Jarek make sense, ie being prescriptive
> on
> > >> what features we want to be there on Day 1 vs launched on 3.x - this
> > does
> > >> allow us to move quickly for our users.
> > >> In line with Constance's concern, we need to be mindful of what we
> > decide
> > >> to keep and what we are willing to cut - which could be a painstaking
> > >> process that could offset any gains of trying to be faster. I also
> > believe
> > >> that bringing all these new amazing features on Airflow 3 will peak
> the
> > >> interest of early adopters and eventually get others interested in
> > >> migration. However, I believe this migration will be a slow process
> and
> > >> will present a gap in certain functionalities that users may want
> before
> > >> entertaining any move to Airflow 3. There are still a lot of folks
> using
> > >> v1.10 today. There were several tactical initiatives in the past few
> > months
> > >> with intent on bringing new functionality, ie Multi-team, to Airflow
> 2.x
> > >> and I feel that while these efforts are not wasted, there should still
> > be
> > >> an option to continue improving Airflow 2 to avoid alienating our
> users
> > on
> > >> the basis of a future promise in Airflow 3, that may not be easy to
> > migrate
> > >> towards.
> > >>
> > >> -- Rajesh
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>> <
> > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > >>
> > >>
> > >> On 2024-05-07, 12:26 PM, "Constance Martineau"
> > >> <constance@astronomer.io.INVA <ma...@astronomer.io.INVA> <mailto:constance@astronomer.io.INVA <ma...@astronomer.io.INVA>>
> > >> <mailto:constance@astronomer.io.INVA <ma...@astronomer.io.INVA> <mailto:
> > constance@astronomer.io.INVA <ma...@astronomer.io.INVA>>>LID>
> > >> wrote:
> > >>
> > >>
> > >>
> > >>
> > >> CAUTION: This email originated from outside of the organization. Do
> not
> > >> click links or open attachments unless you can confirm the sender and
> > know
> > >> the content is safe.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > externe.
> > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> > pouvez
> > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> > que
> > >> le contenu ne présente aucun risque.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> Thank you Jarek for the detailed input. I've taken some time to digest
> > your
> > >> points before responding.
> > >>
> > >>
> > >>
> > >>
> > >> You've outlined a bold vision for Airflow 3, and I agree that being
> > >> decisive about the features and architectures will be the key to
> > success.
> > >> However, before we make final decisions on what features to cut or
> > retain,
> > >> it would be beneficial to have a more comprehensive understanding of
> how
> > >> the current features are utilized by the open-source community.
> > >>
> > >>
> > >>
> > >>
> > >> @Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io> <mailto:kaxil@astronomer.io <ma...@astronomer.io>>
> <mailto:
> > >> kaxil@astronomer.io <ma...@astronomer.io> <mailto:kaxil@astronomer.io <ma...@astronomer.io>>>> recently
> initiated a
> > >> discussion on
> > >> collecting telemetry from open-source deployments:
> > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;> <
> > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;> <
> > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&amp;gt>
> ;>.
> > >> This data
> > >> could be critical in ensuring our decisions are well-informed and
> > reflect
> > >> the real-world usage patterns of our users, not just those from
> managed
> > >> environments like Astro, MWAA or GCC.
> > >>
> > >>
> > >>
> > >>
> > >> It's essential that we challenge our assumptions and base our
> decisions
> > on
> > >> a holistic view of feature usage. Identifying potential cuts is a
> > critical
> > >> step, but let's ensure our strategy aligns with the needs and
> > preferences
> > >> of the broader Airflow community.
> > >>
> > >>
> > >>
> > >>
> > >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>
> <mailto:
> > >> jarek@potiuk.com <ma...@potiuk.com>> <mailto:jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>
> >>>
> > >> wrote:
> > >>
> > >>
> > >>
> > >>
> > >> > I am currently on sick leave, and still recovering - hoping to be
> > able to
> > >> > travel next week to the US as planned, so I just wanted to break out
> > of
> > >> it
> > >> > to make one comment here.
> > >> >
> > >> > I got a clearer head now a bit with medications hopefully working. I
> > am
> > >> > still taking it that should help me to get over the current state,
> > and I
> > >> > wanted to take a look at this discussion unraveling first. Over last
> > week
> > >> > I disconnected from "day-to-day" Airflow and put some thoughts (as
> > much
> > >> as
> > >> > I could in my current state) on it. The whole subject of this thread
> > was
> > >> > started from that - how the current discussions on AIP-67 and others
> > >> change
> > >> > if we consider Airflow 3 is "starting".
> > >> >
> > >> > The price for back-compat is speed of development and quality. More
> > >> > combinations to test, more unexpected issues uncovered, necessity to
> > keep
> > >> > parallel paths (old/new) while adding new features. All what
> Constance
> > >> > wrote about and what Ash explained. We already started to trip over
> > our
> > >> own
> > >> > feet mutliple times in a few last releases. Have we tested all
> > >> combinations
> > >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we
> already
> > see
> > >> > that in a number of "combos" of features things are not working in
> as
> > >> > stable a way as they did before.
> > >> >
> > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a
> > long
> > >> > time (or even move out) as they will not want to move to Airflow 3.
> A
> > lot
> > >> > of the work implemented in AIP-44 and design of AIP-67 was done
> around
> > >> > back-compatibility. but yes -
> > >> > it would have been way easier if designed anew without
> > back-compatibility
> > >> > in mind. And if we implement it and release it in Airflow 2 it will
> > make
> > >> > new Airflow feature development even harder. That's why I wanted to
> > treat
> > >> > it as "tactical" solution - hoping that in Airflow 3 we can make it
> > >> > "properly" - and that's why I started the discussion here when I
> > sensed
> > >> > that we are "close" to Airflow 3 discussion, because I wanted to see
> > what
> > >> > options we have there. This is why I have not yet concluded voting
> on
> > >> > AIP-67 waiting for the result of this discussion here.
> > >> >
> > >> > But if we are ready to go for Airflow.3 then I'd say there are two
> > >> > important things that should be part of the vision.
> > >> >
> > >> > 1) *We should be far more opinionated and have far fewer options of
> > >> > running things in Airflow 3*. Even an order of magnitude more
> > >> opinionated.
> > >> > Make choices, stick to it, perfect those opinionated choices to suit
> > >> 80/20
> > >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not
> > fitting
> > >> > the 20% that might choose to stay at Airflow 2. We can choose now
> > which
> > >> > ~20% of cases we do not want to handle deliberately. And we should
> be
> > >> very,
> > >> > very strict about it. Default should be "no choice". This will
> > radically
> > >> > simplify deployment and should make it easier to simplify Airflow
> > >> > development and DAG authoring experience because we will have less
> > cases
> > >> to
> > >> > support. Even if we plan to add more options in the future, the
> first
> > >> > version of Airflow 3 should support one deployment approach only.
> > This is
> > >> > the only way we can deliver it fast. And we should be very bold
> there.
> > >> > Choose one option and go for it in pretty much every place we have
> > >> choices
> > >> > now. We should Aim for Airflow 3.0 to support only a subset of
> current
> > >> > users - but those who are most likely to migrate first and those
> with
> > the
> > >> > biggest need for the new features. We can think 3.x to support more
> > >> cases,
> > >> > but 3.0 should be as opinionated as humanly possible.
> > >> >
> > >> > And this deployment option should be also something ALL our
> > stakeholders
> > >> > will feel OK with as a way forward in their offering.
> > >> >
> > >> > My candidates (and yes, some are bold):
> > >> >
> > >> > * *Drop MySQL*. If we have a single thing that makes us avoid our
> > schema
> > >> > and DB migration - this is the case. Let's choose Postgres 15+ and
> use
> > >> some
> > >> > of the great features there. This will also enable much faster async
> > SQL
> > >> > implementation and a number of other optimisations - not to mention
> > >> cutting
> > >> > every single change in development and testing time by literally
> half.
> > >> And
> > >> > we should not look back to adding MySQL.
> > >> > * *Drop Celery/Sequential Executor* and start with Local + K8S only
> > (and
> > >> > AWS/Google others can continue developing theirs of course in
> parallel
> > >> and
> > >> > continue Hybrid executor work). Later - we figure out a better
> > solution
> > >> to
> > >> > support "small" tasks using some new K8S features and possibly
> non-k8s
> > >> > solutions (Ray-based?)
> > >> > * *Cut Connection and Variable Management from DB/UI*. Leave only
> > Secrets
> > >> > Management. Later when we have a 100% extensible React UI, we can
> add
> > a
> > >> > "local DB secrets manager" add-on
> > >> > * *Choose a single way for DAG storage that will support versioning
> > from
> > >> > day one*. Bear in mind we can add others later. Bolke's idea of
> using
> > >> > FSspec is an interesting one, we should see if it is feasible.
> > >> > * *Drop FAB completely (including custom plugins) and invest in
> > >> > implementing Auth Manager based on a dedicated, external solution*
> > >> > (KeyCloak
> > >> > that we've discussed before as a likely candidate)
> > >> > * *Leave Providers with Airflow 2 and add tests to make sure they
> are
> > >> > Airflow 3 future-compatible *- develop a way where we continue
> > >> development
> > >> > and contributions for Providers with Airflow 2 and add complete
> tests
> > to
> > >> > run them with Airflow 3. This way we can continue developing
> Provider
> > >> > features independently, and make them work for Airflow 2 (and
> continue
> > >> > adding features for Airflow 2 users alongside Airflow 2 bugfixes),
> > while
> > >> > also gradually fix any Airflow3 incompatibilities and instead of
> > >> > "back-compatibility" tests make provider "forward-compatibility"
> > tests so
> > >> > that future Providers are tested and work on Airflow 3. Also it will
> > make
> > >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested
> without
> > >> > investing in changing the current CI / test harness.
> > >> > * *Simplify Test Harness for Airflow 3 from the start *- without
> > >> providers
> > >> > and 790+ dependencies, we could vastly simplify Airflow3 testing
> > >> (basically
> > >> > make CI jobs from scratch) using mostly standard Python tooling
> > (while we
> > >> > can continue making use of the current test harness for Airflow 2 +
> > >> > Providers and extend it with Airflow 3 future-compatibility tests).
> > That
> > >> > means Breeze would be only staying in Airflow 2 + Providers repo as
> we
> > >> > should be able to achieve most of what we have there with local
> venv/
> > >> > tooling (especially with uv as underlying tooling).
> > >> >
> > >> > 2) *I think we only add very few new "important" features. *Absolute
> > >> > minimum to make Airflow 3 appealing and add them only in Airflow 3:
> > >> > versioning, multi-team, pluggable UI should only be Airflow 3 - it
> > makes
> > >> no
> > >> > sense to invest into Airflow 2 if we already know Airflow 3 is
> coming
> > -
> > >> > that generally triples effort needed to get them out. We should drop
> > new
> > >> > features development in Airflow 2. This will give users incentive to
> > move
> > >> > to 3 if the new features will be worth it. Even paying
> > >> > compatibility/migration price.
> > >> >
> > >> > Versionig, for example: I believe if we decide to go only with
> > Airflow 3
> > >> > and cut some of the above (Postgres only, Single versioning DAG
> > storage)
> > >> we
> > >> > can make bolder decisions in versioning and support simpler models
> > from
> > >> the
> > >> > get go (and deliver it faster). And we should add only a few - but
> > >> > important - features that our users clearly asked for and focus on
> > >> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or
> > >> 2.11).
> > >> > Similarly - multi-team can be simplified if we cut things from the
> > list
> > >> > above and have Task isolation as first-class citizens in Airflow
> (and
> > the
> > >> > only option).
> > >> >
> > >> > My candidates very much concur with the list shared by Kaxil in the
> > doc +
> > >> > I'd add multi-team (but simplified thanks to the cuts). But I also
> > here
> > >> > would mostly revert to Astronomer, Google. AWS team to define
> > >> collectively
> > >> > what is the absolute minimum set of features that would get the
> > "target"
> > >> > part of their customers happy. And ONLY do that.
> > >> >
> > >> > So in short - I think the big part of our discussion should be what
> we
> > >> are
> > >> > ready to drop when we start airflow 3 and be very bold. Once we know
> > we
> > >> > should figure out the absolute minimum of things that we can add
> that
> > >> will
> > >> > benefit a significant part of our users (and make use of increased
> > speed
> > >> > because we dropped things).
> > >> >
> > >> > J.
> > >> >
> > >> >
> > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau
> > >> > <constance@astronomer.io.inva <ma...@astronomer.io.inva> <mailto:constance@astronomer.io.inva <ma...@astronomer.io.inva>>
> > >> <mailto:constance@astronomer.io.inva <ma...@astronomer.io.inva> <mailto:
> > constance@astronomer.io.inva <ma...@astronomer.io.inva>>>lid>
> > >> wrote:
> > >> >
> > >> > > Hi Michal,
> > >> > >
> > >> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate
> > your
> > >> > > concerns about the migration overhead for our users with a major
> new
> > >> > > version and see the appeal in your suggestion to integrate many of
> > the
> > >> > > proposed changes into Airflow 2 through separate AIPs. It’s a
> valid
> > >> point
> > >> > > and certainly aligns with the value of making incremental
> > improvements.
> > >> > >
> > >> > > However, after looking closely at the enhancements outlined for
> > Airflow
> > >> > 3,
> > >> > > I'm convinced they warrant a new major release. Here’s why:
> > >> > >
> > >> > > 1. *Core Architectural Changes:* We’re looking at foundational
> > changes
> > >> > > with Airflow 3—like redefining task priorities, separating task
> > >> > > definition
> > >> > > and task execution, and new AIPs like DAG versioning. remote
> > execution
> > >> > > and restricting database access from workers. These aren’t just
> > >> > > incremental
> > >> > > improvements but major shifts that will set the stage for the next
> > >> > > decade
> > >> > > of Airflow’s architecture. Grouping these changes into a major
> > release
> > >> > > will
> > >> > > help us make these transitions more cleanly and with fewer
> > constraints
> > >> > > from
> > >> > > past decisions.
> > >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140
> > >> > deprecated
> > >> > > issues, and this will only grow if we continue without a major
> > >> > cleanup.
> > >> > > This makes it increasingly difficult to implement new features
> > >> > > effectively
> > >> > > while maintaining backward compatibility. A major release allows
> us
> > to
> > >> > > address these issues head-on, reducing technical debt and paving
> the
> > >> > way
> > >> > > for a more robust platform.
> > >> > > 3. *Managing Breaking Changes:* Let’s take the example of
> > restricting
> > >> > > database access from workers. It’s a necessary move for better
> > >> > security
> > >> > > and
> > >> > > also potentially scalability reasons (reduces DB load). Many users
> > >> > have
> > >> > > workflows that interact with the DB, either by using raw sql or by
> > >> > > leveraging a session object. We could implement this feature in
> > >> > Airflow
> > >> > > 2
> > >> > > and avoid breaking existing workflows by continuing to have the
> old
> > >> > > standard mode as default - much of the work is already done - but
> > that
> > >> > > would mean supporting both the new secure mode and the old
> standard
> > >> > mode
> > >> > > indefinitely and design new features with the assumption that most
> > >> > will
> > >> > > continue using the old standard mode. With Airflow 3, we can make
> > >> > secure
> > >> > > mode the default or even the only option, simplifying
> implementation
> > >> > and
> > >> > > future development. This is just one example where it is feasible
> to
> > >> > > implement in Airflow 2, but is better if we release it under the
> > >> > > context of
> > >> > > Airflow 3.
> > >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up
> > >> > > possibilities for handling workflows beyond batch processing.
> > Features
> > >> > > like
> > >> > > real-time DAG execution through API and multi-language task
> support
> > >> > are
> > >> > > big
> > >> > > steps forward, significantly expanding Airflow’s utility.
> > >> > >
> > >> > >
> > >> > > While integrating these updates into Airflow 2 might look less
> > >> disruptive
> > >> > > initially, the scale and nature of the required changes really
> > support
> > >> a
> > >> > > move to Airflow 3. It’s not just about adding new features; it’s
> > about
> > >> > > setting up Airflow so that it continues to remain relevant for the
> > next
> > >> > ten
> > >> > > years.
> > >> > >
> > >> > > Constance
> > >> > >
> > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>
> > >> <mailto:ash@apache.org <ma...@apache.org>> <mailto:ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <ma...@apache.org>
> > >>>
> > >> wrote:
> > >> > >
> > >> > > > There's a lot of technical debt hiding in Airflow, especially
> the
> > >> > > > scheduler that makes it harder and harder to efficiently add new
> > >> > > features.
> > >> > > >
> > >> > > > At some point, very soon, we are going to have to remove some
> very
> > >> > > > infrequently used back compat shims that negatively affect
> > >> performance.
> > >> > > > Without doing that the pace at which we can realistically add
> > some of
> > >> > the
> > >> > > > more exciting features tends towards zero. Developer speed of
> > >> > > contributors
> > >> > > > is a factor here too!
> > >> > > >
> > >> > > > So while we are still using SemVer, that necessitates v3.
> > >> > > >
> > >> > > > Ash
> > >> > > >
> > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <
> > michalmodras@google.com <ma...@google.com>
> > >> <mailto:michalmodras@google.com <ma...@google.com>> <mailto:michalmodras@google.com <ma...@google.com>
> > <mailto:
> > >> michalmodras@google.com <ma...@google.com>>>
> > >> > > .INVALID>
> > >> > > > wrote:
> > >> > > > >+1 to Jens's & Bolke's points here and in the doc
> > >> > > > >
> > >> > > > >I agree we should work on clarifying the directions we would
> like
> > >> > > Airflow
> > >> > > > >to go. Introducing a new major Airflow version is a massive
> > overhead
> > >> > for
> > >> > > > >users, who would need to plan for migrations, onboarding the
> new
> > >> > Airflow
> > >> > > > >(with a slightly different architecture), etc., and effectively
> > >> > Airflow
> > >> > > 2
> > >> > > > >would live in parallel for a long time.
> > >> > > > >
> > >> > > > >Personally, I think most of the points in Kaxil's/Vikram's doc
> > are
> > >> > > > valuable
> > >> > > > >projects of their own, and I could imagine all of them being
> > >> delivered
> > >> > > as
> > >> > > > >separate AIPs within Airflow 2 (surely new minor versions of
> > Airflow
> > >> > > 2). I
> > >> > > > >am not sure if the scope of changes and the goal we want to
> > achieve
> > >> is
> > >> > > a)
> > >> > > > >clear enough b) broad enough to call for a new major version.
> > >> > > > >
> > >> > > > >Best,
> > >> > > > >Michal
> > >> > > > >
> > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens
> (XC-AS/EAE-ADA-T)
> > >> > > > ><Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva> <mailto:
> > >> Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva>> <mailto:
> > Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva>
> > >> <mailto:Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva>>>lid> wrote:
> > >> > > > >
> > >> > > > >> Thanks for the document write-up, Kaxil. I assume this is
> > mostly a
> > >> > > > vision
> > >> > > > >> statement.
> > >> > > > >>
> > >> > > > >> Looking forward for a larger addendum where we can collect
> > things
> > >> > that
> > >> > > > we
> > >> > > > >> all can vote and agree on as targets.
> > >> > > > >>
> > >> > > > >> As I started earlier with a confluence page and it seems this
> > is
> > >> not
> > >> > > > >> accessible to all, shall we convert this to a Google Doc for
> > >> better
> > >> > > > >> collaboration and item collection?
> > >> > > > >>
> > >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> <https://aka.ms/o0ukef&gt;> <
> > >> https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&amp;gt;&gt;> <https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&amp;gt;&gt;> <
> > >> https://aka.ms/o0ukef&amp;gt;&gt;> <https://aka.ms/o0ukef&amp;amp;gt;&amp;gt;&gt;>
> > >> > > > >> ________________________________
> > >> > > > >> From: Vikram Koka <vikram@astronomer.io.INVA <ma...@astronomer.io.INVA> <mailto:
> > >> vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>> <mailto:vikram@astronomer.io.INVA <ma...@astronomer.io.INVA> <mailto:
> > >> vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>>>LID>
> > >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> > >> > > > >> To: dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>
> > >> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>> <
> > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>>>
> > >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow
> > 2)
> > >> vs
> > >> > > > >> strategic (Airflow 3) approach
> > >> > > > >>
> > >> > > > >> Thank you for your feedback, Bolke and Andrey!
> > >> > > > >>
> > >> > > > >> Bolke,
> > >> > > > >> I have replied to some of your comments in the doc.
> > >> > > > >> I will provide a detailed write up on the "Interactive DAG
> run"
> > >> (or
> > >> > > > >> synchronous DAG run) capability, which has generated some
> early
> > >> > > > questions.
> > >> > > > >> I had intended to get an AIP published for that as a
> follow-up,
> > >> but
> > >> > I
> > >> > > > >> believe that a simpler write up would be useful ahead of the
> > AIP.
> > >> > > > >>
> > >> > > > >> Andrey,
> > >> > > > >> You raise an interesting point.
> > >> > > > >>
> > >> > > > >> As part of the Airflow 2.0 release, we as a community had
> > decided
> > >> to
> > >> > > > >> strictly adhere to Semver as detailed in the document you
> > >> > referenced.
> > >> > > We
> > >> > > > >> also consciously split out the "Core Airflow" releases from
> the
> > >> > > > "Provider"
> > >> > > > >> releases at that time. We had a clear expectation then for
> the
> > >> > cadence
> > >> > > > of
> > >> > > > >> both minor and patch releases, which we have generally
> adhered
> > to
> > >> > > since
> > >> > > > >> then.
> > >> > > > >>
> > >> > > > >> Personally, I am more concerned about our Provider releases
> > right
> > >> > now,
> > >> > > > as
> > >> > > > >> compared to the cadence of our major releases. I believe that
> > one
> > >> of
> > >> > > the
> > >> > > > >> proposed changes in the Airflow 3 document i.e. the clear
> > >> separation
> > >> > > for
> > >> > > > >> Task Execution will help here, but more may be needed.
> > >> > > > >>
> > >> > > > >> Definitely interested in more feedback on this as well.
> > >> > > > >>
> > >> > > > >> Vikram
> > >> > > > >>
> > >> > > > >>
> > >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> > >> > > andrey.anshin@taragol.is <ma...@taragol.is> <mailto:andrey.anshin@taragol.is <ma...@taragol.is>>
> <mailto:
> > >> andrey.anshin@taragol.is <ma...@taragol.is> <mailto:andrey.anshin@taragol.is <ma...@taragol.is>>>
> > >> > > > >
> > >> > > > >> wrote:
> > >> > > > >>
> > >> > > > >> > I would like to propose to change (at least discuss)
> release
> > >> > policy
> > >> > > > >> around
> > >> > > > >> > the Major version of Airflow.
> > >> > > > >> >
> > >> > > > >> > Right now it is described as "These releases do not happen
> > with
> > >> > any
> > >> > > > >> regular
> > >> > > > >> > interval or on any predictable schedule." :
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0>
> > >> <
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0 <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0>
> > >
> > >> <
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0 <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0>
> > >
> > >> <
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;amp;reserved=0&amp;gt>
> > >> ;>
> > >> > > > >> <
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release>
> > >> <
> > >>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release>
> > >
> > >> <
> > >>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release>
> > >
> > >> <
> > >>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&amp;gt>
> > >> ;>
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > So maybe it is time to make it schedulable, e.g. one per
> two
> > >> years
> > >> > > or
> > >> > > > so.
> > >> > > > >> > This one could help us to avoid such a discussion in the
> > future,
> > >> > > like
> > >> > > > "We
> > >> > > > >> > don't know when Airflow 4 is coming.". At the moment when
> the
> > >> new
> > >> > > > major
> > >> > > > >> > version will be released new features wouldn't be added in
> > the
> > >> old
> > >> > > > major
> > >> > > > >> > version, however we would support bug / security for a
> while,
> > >> > e.g. 1
> > >> > > > year
> > >> > > > >> > for bug fixes, 3 years for security fixes with a total 5
> year
> > >> > > > lifecycle
> > >> > > > >> per
> > >> > > > >> > a major version. These just are approximate time periods
> for
> > a
> > >> > > > definition
> > >> > > > >> > of current period, bugfix period and security fix period.
> > >> > > > >> >
> > >> > > > >> > In contributors' perspective it helps with dropping the
> > >> deprecated
> > >> > > > stuff
> > >> > > > >> > which resolves some old problem: we have to support
> > everything
> > >> > > > including
> > >> > > > >> > deprecated stuff and without schedulable lifecycle for the
> > >> > > deprecated
> > >> > > > >> stuff
> > >> > > > >> > it could be showstopper for the new feature, because
> > sometimes
> > >> it
> > >> > > > hard to
> > >> > > > >> > support two different approaches for long period of time
> > with no
> > >> > > hope
> > >> > > > >> that
> > >> > > > >> > it will happen soon. For some fundamental stuff which do
> not
> > >> > > require a
> > >> > > > >> lot
> > >> > > > >> > things time to support we could postponed removal for next
> > after
> > >> > the
> > >> > > > next
> > >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in
> > Airflow 5
> > >> > > > >> >
> > >> > > > >> > In the user perspective, they have at least bug fix support
> > for
> > >> a
> > >> > > > while,
> > >> > > > >> if
> > >> > > > >> > someone want to use legacy version it their choice, however
> > no
> > >> new
> > >> > > > >> > features, no new version of providers (after one year)
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > ----
> > >> > > > >> > Best Wishes
> > >> > > > >> > *Andrey Anshin*
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <
> > bdbruin@gmail.com <ma...@gmail.com>
> > >> <mailto:bdbruin@gmail.com <ma...@gmail.com>> <mailto:bdbruin@gmail.com <ma...@gmail.com> <mailto:
> > >> bdbruin@gmail.com <ma...@gmail.com>>>>
> > >> > > > wrote:
> > >> > > > >> >
> > >> > > > >> > > I have left several comments :-). And on interactive dag
> > runs
> > >> > even
> > >> > > > >> after
> > >> > > > >> > > the explanation of Vikram I still don't have a clue what
> we
> > >> want
> > >> > > to
> > >> > > > >> > > accomplish there :-P.
> > >> > > > >> > >
> > >> > > > >> > > I would like to see a mantra or team for Airflow 3. That
> > helps
> > >> > > > nudging
> > >> > > > >> > > people in the same direction. Suggestions in the
> comments.
> > >> > > > >> > >
> > >> > > > >> > > Bolke
> > >> > > > >> > > Sent from my iPhone
> > >> > > > >> > >
> > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> > >> > > <vikram@astronomer.io.inva <ma...@astronomer.io.inva> <mailto:vikram@astronomer.io.inva <ma...@astronomer.io.inva>>
> > <mailto:
> > >> vikram@astronomer.io.inva <ma...@astronomer.io.inva> <mailto:vikram@astronomer.io.inva <ma...@astronomer.io.inva>>>lid
> > >> > > > >
> > >> > > > >> > > wrote:
> > >> > > > >> > > >
> > >> > > > >> > > > Good point Jed.
> > >> > > > >> > > > I responded back to your comment in the doc as well and
> > very
> > >> > > open
> > >> > > > to
> > >> > > > >> > > > changing the term in the doc.
> > >> > > > >> > > >
> > >> > > > >> > > > Used the term "interactive DAG run" as the ability to
> > invoke
> > >> > or
> > >> > > > >> > trigger a
> > >> > > > >> > > > DAG run through the API, with the expectation of
> getting
> > >> back
> > >> > a
> > >> > > > >> result
> > >> > > > >> > > > immediately. An alternate term could be a "synchronous
> > DAG
> > >> > run".
> > >> > > > >> > > >
> > >> > > > >> > > > Regardless, this is a significant change so a good term
> > to
> > >> > > > indicate
> > >> > > > >> the
> > >> > > > >> > > > expansion from "batch runs only" is warranted. Very
> open
> > to
> > >> > > > different
> > >> > > > >> > > terms
> > >> > > > >> > > > here.
> > >> > > > >> > > >
> > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > >> > > > >> > jedcunningham@apache.org <ma...@apache.org> <mailto:jedcunningham@apache.org <ma...@apache.org>>
> > >> <mailto:jedcunningham@apache.org <ma...@apache.org> <mailto:jedcunningham@apache.org <ma...@apache.org>>>
> > >> > > > >> > > >
> > >> > > > >> > > >> wrote:
> > >> > > > >> > > >>
> > >> > > > >> > > >> Very exciting! Looks like we will have a busy period
> of
> > >> time
> > >> > > > ahead
> > >> > > > >> of
> > >> > > > >> > > us.
> > >> > > > >> > > >> Overall I like the plan so far, especially using this
> > >> year's
> > >> > > > Airflow
> > >> > > > >> > > Summit
> > >> > > > >> > > >> as an opportunity to announce and gather feedback, and
> > the
> > >> > 2025
> > >> > > > >> > version
> > >> > > > >> > > to
> > >> > > > >> > > >> pitch upgrading.
> > >> > > > >> > > >>
> > >> > > > >> > > >> I left a comment in the doc, but we might want to
> > iterate
> > >> on
> > >> > > the
> > >> > > > >> > > >> terminology we use for high priority or "synchronous"
> > DAG
> > >> > runs
> > >> > > to
> > >> > > > >> > serve
> > >> > > > >> > > LLM
> > >> > > > >> > > >> responses - I find "interactive DAG runs" a bit
> > confusing.
> > >> > > > >> > > >>
> > >> > > > >> > >
> > >> > > > >> > >
> > >> > > >
> > ---------------------------------------------------------------------
> > >> > > > >> > > To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>
> > >> <mailto:dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > >> dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > >> dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>>>
> > >> > > > >> > > For additional commands, e-mail:
> > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > >> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>> <mailto:
> > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > >> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>>>
> > >> > > > >> > >
> > >> > > > >> > >
> > >> > > > >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Pierre Jeambrun <pi...@gmail.com>]

Thank you for the work and proposal made around airflow 3.

I also strongly believe that those important changes need to happen for
airflow to stay relevant and it would be the right moment to start
discussing and planning for airflow 3. As mentioned some features are hard
to develop within airflow 2 with backward compatibility in mind:
 * I would love to see a full react front end, without FAB.
* We could also take the opportunity to rework and clarify our RBAC code,
we had quite a number of CVEs related to it.
* Do the breaking change for api clients and (finally) upgrade the openapi
client generator to a more recent version
* Extract business service layers that could be shared across the entire
codebase instead of having a 'component wise' approach. Such that handling
of dag dates (execution, logical), serialization, resource access control
and such tools could be implemented once and shared everywhere. (easier
maintenance, more consistency between API, CLI, and other core components
behaviors). Inspired by some famous software architecture such as hexagonal
or clean. Even if I don't think that those would be relevant to airflow
codebase stricto sensu.

As mentioned stability for our users is key and supporting airflow 2.x for
a while after airflow 3 is a must have. I just want to bring to
your attention that this means more work for our release managers and also
for the security team (backporting security patches, CVEs handling for two
different major versions, etc.). All that to say that we should choose this
period wisely. (long enough, but not too long)

This sounds like a really exciting time for airflow.
Cheers

Le jeu. 9 mai 2024 à 11:23, Amogh Desai <am...@gmail.com> a écrit :

> Thanks for initiating this discussion @Kaxil Naik <ka...@astronomer.io>!
>
>
> Read through the document and the emails here and I like the direction in
> which
> we are proceeding.
>
> I also agree that removing code is almost as important as adding code. We
> need to have a clear checklist of what we want to
> remove and get a vote for the same from the community.
>
> I also see that we have multiple ways of doing a thing, which is not needed
> (serialising for example) and removing
> which can lead to simpler interfaces for users, and contributors too.
>
> I also find the *secret masker* overly complex personally, why can't it be
> simpler :)
>
> However, I have a small concern with the proportion of the users that would
> like to migrate to an early version of
> Airflow 3 if we do not support various deployment modes. If we dont get
> enough initial feedback, we do not know how the new
> major is doing, no?
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > Strong agree - I think one of the things I regret most about Airflow 2
> was
> > that we didn't remove enough. It probably eased the upgrade process for
> > users though.
> >
> > > but I was hoping one day when we remove providers
> > > from main development, I can do it personally and beat Ash to it.
> >
> > Grrr ;l
> >
> > The other thing I want to look at (but don't have concrete ideas of yet)
> > is some of the lesser used dag/task concurrency and dependency features -
> > if we can remove a few things and get a faster scheduler than it's worth
> > having a discussion over.
> >
> > On 9 May 2024 06:31:38 BST, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >I hope more voices will be coming soon as well :).
> > >
> > >Constance - few words of explanation, for my "remove" things. I do not
> say
> > >we "have to" remove those things or even that we "should".
> > >
> > >My main motivation for mentioning the list is that we often forget that
> > >when we have a new major version of software - removing stuff is as
> > >important as adding. So I think when it comes to final decisions about
> > >Airflow 3 we should be well aware about both: what we add and what we
> > >remove. So my main point is - "removal" of things should be as
> important a
> > >part in our future discussions as "adding", and I'd say with 10 years of
> > >Airflow history, it's more important to remove things than to add them.
> > >
> > >Side comment: Ash for one mentioned at multiple occasions how proud he
> is
> > >that his contributions to Airflow are net-negative - he removed more
> code
> > >than added. Unfortunately it's not shown any more here
> > >https://github.com/apache/airflow/graphs/contributors - because we have
> > >more than 10.000 commits, but I was hoping one day when we remove
> > providers
> > >from main development, I can do it personally and beat Ash to it.
> > >
> > >Just wanted to stress a few things on why I think "discussing potential
> > >removals" is important:
> > >
> > >* I think it is super important to deliver Airflow 3 really fast. It
> took
> > >
> > >1.5 year to release Airflow 2 and it was far too long and keeping
> Airflow
> > 1
> > >and 2 in parallel was a huge pain and drain and it slowed us down
> > >enormously. We should absolutely limit the time when we actively develop
> > >both Airflow 2 features and Airflow 3 features.
> > >* I think most of the current deprecations of the 100+ we have does not
> > >slow us down AT ALL. Removal of those is mostly cosmetic change, a
> little
> > >bit clutter to remove but they have little-to-no impact on actual speed
> of
> > >development, it's just an old code that keeps on being around
> > >* on the other hand - some of the things I mentioned ARE slowing us
> down A
> > >LOT and will continue to do so until we remove them. For example, I
> > believe
> > >"postgresql + mysql + sqlite complete versioning solution for all the
> > >possible variants of storage" will be quite a bit more complex and
> testing
> > >will take far more time than if we drop mysql and choose a single
> storage
> > >approach for Airflow 3.0. I have no hard data to back it up, but my gut
> > >feeling is that it can take at least twice as long to get it out in the
> > >hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for
> > >all the options we have there.
> > >* the telemetry will be cool - but even if we add it for 2.10, the first
> > >time meaningful data will be available is maybe 2 years from now when we
> > do
> > >Airflow 4. For now we can completely forget about it because it's only
> > >going to show us some early adopters of Airflow 2.10. But we have
> surveys
> > +
> > >we can collect data from Astronomer, Google. Amazon for some usage and
> > >identify who will be early adopters of Airflow 3 and target them first.
> > >* the fact that we will drop something for Airflow 3.0, does not mean
> that
> > >we have to drop it "forever". We can safely assume that Airflow 3.0 will
> > be
> > >only used by early adopters, and many of our users will wait for 3.1,
> 3.2
> > >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those
> "slower
> > >moving users" from the 3.0. We can re-add things in 3.1 building on
> > >foundations of 3.0 after it is in the hands of those early adopters. I
> > >personally think we should prioritize speed of delivery of 3.0 over
> > >"complete support of every deployment option of Airflow 2" - to allow
> > >**some** of our early adopter users to migrate quickly, and add what's
> > >missing for those who would like to move later in later versions.
> > >* I think - this is the most important "product management" decision
> here
> > -
> > >based on the data we have available. Which users do we target for
> Airflow
> > >3.0, and what we can drop to deliver it faster (and possibly re-add more
> > >stuff for 3.1+). I think it's crucial to the success of the Airflow 3.0
> > >initiative and the most important decision from the Product management
> > side
> > >that will impact the timeline of Airflow 3.0. While a lot depends on how
> > >much time and effort individuals will spend on implementing things, the
> > >decision of what we drop will impact everyone's speed and overall
> delivery
> > >of Airflow 3.0. It's a decision we should not take lightly.
> > >
> > >So my main point is: let's add "what we remove" as a very important
> point
> > >in the product discussions we are going to have.
> > >
> > >J.
> > >
> > >
> > >
> > >
> > >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar
> > ><rb...@amazon.com.invalid> wrote:
> > >
> > >> Like Constance and many others, there was time needed to process this
> > >> fantastical summary and approach to Airflow 3. __
> > >> A lot of the points raised by Jarek make sense, ie being prescriptive
> on
> > >> what features we want to be there on Day 1 vs launched on 3.x - this
> > does
> > >> allow us to move quickly for our users.
> > >> In line with Constance's concern, we need to be mindful of what we
> > decide
> > >> to keep and what we are willing to cut - which could be a painstaking
> > >> process that could offset any gains of trying to be faster. I also
> > believe
> > >> that bringing all these new amazing features on Airflow 3 will peak
> the
> > >> interest of early adopters and eventually get others interested in
> > >> migration. However, I believe this migration will be a slow process
> and
> > >> will present a gap in certain functionalities that users may want
> before
> > >> entertaining any move to Airflow 3. There are still a lot of folks
> using
> > >> v1.10 today. There were several tactical initiatives in the past few
> > months
> > >> with intent on bringing new functionality, ie Multi-team, to Airflow
> 2.x
> > >> and I feel that while these efforts are not wasted, there should still
> > be
> > >> an option to continue improving Airflow 2 to avoid alienating our
> users
> > on
> > >> the basis of a future promise in Airflow 3, that may not be easy to
> > migrate
> > >> towards.
> > >>
> > >> -- Rajesh
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> dev@airflow.apache.org <ma...@airflow.apache.org> <
> > >> dev@airflow.apache.org <ma...@airflow.apache.org>>
> > >>
> > >>
> > >> On 2024-05-07, 12:26 PM, "Constance Martineau"
> > >> <constance@astronomer.io.INVA <ma...@astronomer.io.INVA>
> > >> <mailto:constance@astronomer.io.INVA <mailto:
> > constance@astronomer.io.INVA>>LID>
> > >> wrote:
> > >>
> > >>
> > >>
> > >>
> > >> CAUTION: This email originated from outside of the organization. Do
> not
> > >> click links or open attachments unless you can confirm the sender and
> > know
> > >> the content is safe.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > externe.
> > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> > pouvez
> > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> > que
> > >> le contenu ne présente aucun risque.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> Thank you Jarek for the detailed input. I've taken some time to digest
> > your
> > >> points before responding.
> > >>
> > >>
> > >>
> > >>
> > >> You've outlined a bold vision for Airflow 3, and I agree that being
> > >> decisive about the features and architectures will be the key to
> > success.
> > >> However, before we make final decisions on what features to cut or
> > retain,
> > >> it would be beneficial to have a more comprehensive understanding of
> how
> > >> the current features are utilized by the open-source community.
> > >>
> > >>
> > >>
> > >>
> > >> @Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io>
> <mailto:
> > >> kaxil@astronomer.io <ma...@astronomer.io>>> recently
> initiated a
> > >> discussion on
> > >> collecting telemetry from open-source deployments:
> > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m <
> > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> > >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt
> ;>.
> > >> This data
> > >> could be critical in ensuring our decisions are well-informed and
> > reflect
> > >> the real-world usage patterns of our users, not just those from
> managed
> > >> environments like Astro, MWAA or GCC.
> > >>
> > >>
> > >>
> > >>
> > >> It's essential that we challenge our assumptions and base our
> decisions
> > on
> > >> a holistic view of feature usage. Identifying potential cuts is a
> > critical
> > >> step, but let's ensure our strategy aligns with the needs and
> > preferences
> > >> of the broader Airflow community.
> > >>
> > >>
> > >>
> > >>
> > >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <jarek@potiuk.com
> <mailto:
> > >> jarek@potiuk.com> <mailto:jarek@potiuk.com <mailto:jarek@potiuk.com
> >>>
> > >> wrote:
> > >>
> > >>
> > >>
> > >>
> > >> > I am currently on sick leave, and still recovering - hoping to be
> > able to
> > >> > travel next week to the US as planned, so I just wanted to break out
> > of
> > >> it
> > >> > to make one comment here.
> > >> >
> > >> > I got a clearer head now a bit with medications hopefully working. I
> > am
> > >> > still taking it that should help me to get over the current state,
> > and I
> > >> > wanted to take a look at this discussion unraveling first. Over last
> > week
> > >> > I disconnected from "day-to-day" Airflow and put some thoughts (as
> > much
> > >> as
> > >> > I could in my current state) on it. The whole subject of this thread
> > was
> > >> > started from that - how the current discussions on AIP-67 and others
> > >> change
> > >> > if we consider Airflow 3 is "starting".
> > >> >
> > >> > The price for back-compat is speed of development and quality. More
> > >> > combinations to test, more unexpected issues uncovered, necessity to
> > keep
> > >> > parallel paths (old/new) while adding new features. All what
> Constance
> > >> > wrote about and what Ash explained. We already started to trip over
> > our
> > >> own
> > >> > feet mutliple times in a few last releases. Have we tested all
> > >> combinations
> > >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we
> already
> > see
> > >> > that in a number of "combos" of features things are not working in
> as
> > >> > stable a way as they did before.
> > >> >
> > >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a
> > long
> > >> > time (or even move out) as they will not want to move to Airflow 3.
> A
> > lot
> > >> > of the work implemented in AIP-44 and design of AIP-67 was done
> around
> > >> > back-compatibility. but yes -
> > >> > it would have been way easier if designed anew without
> > back-compatibility
> > >> > in mind. And if we implement it and release it in Airflow 2 it will
> > make
> > >> > new Airflow feature development even harder. That's why I wanted to
> > treat
> > >> > it as "tactical" solution - hoping that in Airflow 3 we can make it
> > >> > "properly" - and that's why I started the discussion here when I
> > sensed
> > >> > that we are "close" to Airflow 3 discussion, because I wanted to see
> > what
> > >> > options we have there. This is why I have not yet concluded voting
> on
> > >> > AIP-67 waiting for the result of this discussion here.
> > >> >
> > >> > But if we are ready to go for Airflow.3 then I'd say there are two
> > >> > important things that should be part of the vision.
> > >> >
> > >> > 1) *We should be far more opinionated and have far fewer options of
> > >> > running things in Airflow 3*. Even an order of magnitude more
> > >> opinionated.
> > >> > Make choices, stick to it, perfect those opinionated choices to suit
> > >> 80/20
> > >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not
> > fitting
> > >> > the 20% that might choose to stay at Airflow 2. We can choose now
> > which
> > >> > ~20% of cases we do not want to handle deliberately. And we should
> be
> > >> very,
> > >> > very strict about it. Default should be "no choice". This will
> > radically
> > >> > simplify deployment and should make it easier to simplify Airflow
> > >> > development and DAG authoring experience because we will have less
> > cases
> > >> to
> > >> > support. Even if we plan to add more options in the future, the
> first
> > >> > version of Airflow 3 should support one deployment approach only.
> > This is
> > >> > the only way we can deliver it fast. And we should be very bold
> there.
> > >> > Choose one option and go for it in pretty much every place we have
> > >> choices
> > >> > now. We should Aim for Airflow 3.0 to support only a subset of
> current
> > >> > users - but those who are most likely to migrate first and those
> with
> > the
> > >> > biggest need for the new features. We can think 3.x to support more
> > >> cases,
> > >> > but 3.0 should be as opinionated as humanly possible.
> > >> >
> > >> > And this deployment option should be also something ALL our
> > stakeholders
> > >> > will feel OK with as a way forward in their offering.
> > >> >
> > >> > My candidates (and yes, some are bold):
> > >> >
> > >> > * *Drop MySQL*. If we have a single thing that makes us avoid our
> > schema
> > >> > and DB migration - this is the case. Let's choose Postgres 15+ and
> use
> > >> some
> > >> > of the great features there. This will also enable much faster async
> > SQL
> > >> > implementation and a number of other optimisations - not to mention
> > >> cutting
> > >> > every single change in development and testing time by literally
> half.
> > >> And
> > >> > we should not look back to adding MySQL.
> > >> > * *Drop Celery/Sequential Executor* and start with Local + K8S only
> > (and
> > >> > AWS/Google others can continue developing theirs of course in
> parallel
> > >> and
> > >> > continue Hybrid executor work). Later - we figure out a better
> > solution
> > >> to
> > >> > support "small" tasks using some new K8S features and possibly
> non-k8s
> > >> > solutions (Ray-based?)
> > >> > * *Cut Connection and Variable Management from DB/UI*. Leave only
> > Secrets
> > >> > Management. Later when we have a 100% extensible React UI, we can
> add
> > a
> > >> > "local DB secrets manager" add-on
> > >> > * *Choose a single way for DAG storage that will support versioning
> > from
> > >> > day one*. Bear in mind we can add others later. Bolke's idea of
> using
> > >> > FSspec is an interesting one, we should see if it is feasible.
> > >> > * *Drop FAB completely (including custom plugins) and invest in
> > >> > implementing Auth Manager based on a dedicated, external solution*
> > >> > (KeyCloak
> > >> > that we've discussed before as a likely candidate)
> > >> > * *Leave Providers with Airflow 2 and add tests to make sure they
> are
> > >> > Airflow 3 future-compatible *- develop a way where we continue
> > >> development
> > >> > and contributions for Providers with Airflow 2 and add complete
> tests
> > to
> > >> > run them with Airflow 3. This way we can continue developing
> Provider
> > >> > features independently, and make them work for Airflow 2 (and
> continue
> > >> > adding features for Airflow 2 users alongside Airflow 2 bugfixes),
> > while
> > >> > also gradually fix any Airflow3 incompatibilities and instead of
> > >> > "back-compatibility" tests make provider "forward-compatibility"
> > tests so
> > >> > that future Providers are tested and work on Airflow 3. Also it will
> > make
> > >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested
> without
> > >> > investing in changing the current CI / test harness.
> > >> > * *Simplify Test Harness for Airflow 3 from the start *- without
> > >> providers
> > >> > and 790+ dependencies, we could vastly simplify Airflow3 testing
> > >> (basically
> > >> > make CI jobs from scratch) using mostly standard Python tooling
> > (while we
> > >> > can continue making use of the current test harness for Airflow 2 +
> > >> > Providers and extend it with Airflow 3 future-compatibility tests).
> > That
> > >> > means Breeze would be only staying in Airflow 2 + Providers repo as
> we
> > >> > should be able to achieve most of what we have there with local
> venv/
> > >> > tooling (especially with uv as underlying tooling).
> > >> >
> > >> > 2) *I think we only add very few new "important" features. *Absolute
> > >> > minimum to make Airflow 3 appealing and add them only in Airflow 3:
> > >> > versioning, multi-team, pluggable UI should only be Airflow 3 - it
> > makes
> > >> no
> > >> > sense to invest into Airflow 2 if we already know Airflow 3 is
> coming
> > -
> > >> > that generally triples effort needed to get them out. We should drop
> > new
> > >> > features development in Airflow 2. This will give users incentive to
> > move
> > >> > to 3 if the new features will be worth it. Even paying
> > >> > compatibility/migration price.
> > >> >
> > >> > Versionig, for example: I believe if we decide to go only with
> > Airflow 3
> > >> > and cut some of the above (Postgres only, Single versioning DAG
> > storage)
> > >> we
> > >> > can make bolder decisions in versioning and support simpler models
> > from
> > >> the
> > >> > get go (and deliver it faster). And we should add only a few - but
> > >> > important - features that our users clearly asked for and focus on
> > >> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or
> > >> 2.11).
> > >> > Similarly - multi-team can be simplified if we cut things from the
> > list
> > >> > above and have Task isolation as first-class citizens in Airflow
> (and
> > the
> > >> > only option).
> > >> >
> > >> > My candidates very much concur with the list shared by Kaxil in the
> > doc +
> > >> > I'd add multi-team (but simplified thanks to the cuts). But I also
> > here
> > >> > would mostly revert to Astronomer, Google. AWS team to define
> > >> collectively
> > >> > what is the absolute minimum set of features that would get the
> > "target"
> > >> > part of their customers happy. And ONLY do that.
> > >> >
> > >> > So in short - I think the big part of our discussion should be what
> we
> > >> are
> > >> > ready to drop when we start airflow 3 and be very bold. Once we know
> > we
> > >> > should figure out the absolute minimum of things that we can add
> that
> > >> will
> > >> > benefit a significant part of our users (and make use of increased
> > speed
> > >> > because we dropped things).
> > >> >
> > >> > J.
> > >> >
> > >> >
> > >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau
> > >> > <constance@astronomer.io.inva <ma...@astronomer.io.inva>
> > >> <mailto:constance@astronomer.io.inva <mailto:
> > constance@astronomer.io.inva>>lid>
> > >> wrote:
> > >> >
> > >> > > Hi Michal,
> > >> > >
> > >> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate
> > your
> > >> > > concerns about the migration overhead for our users with a major
> new
> > >> > > version and see the appeal in your suggestion to integrate many of
> > the
> > >> > > proposed changes into Airflow 2 through separate AIPs. It’s a
> valid
> > >> point
> > >> > > and certainly aligns with the value of making incremental
> > improvements.
> > >> > >
> > >> > > However, after looking closely at the enhancements outlined for
> > Airflow
> > >> > 3,
> > >> > > I'm convinced they warrant a new major release. Here’s why:
> > >> > >
> > >> > > 1. *Core Architectural Changes:* We’re looking at foundational
> > changes
> > >> > > with Airflow 3—like redefining task priorities, separating task
> > >> > > definition
> > >> > > and task execution, and new AIPs like DAG versioning. remote
> > execution
> > >> > > and restricting database access from workers. These aren’t just
> > >> > > incremental
> > >> > > improvements but major shifts that will set the stage for the next
> > >> > > decade
> > >> > > of Airflow’s architecture. Grouping these changes into a major
> > release
> > >> > > will
> > >> > > help us make these transitions more cleanly and with fewer
> > constraints
> > >> > > from
> > >> > > past decisions.
> > >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140
> > >> > deprecated
> > >> > > issues, and this will only grow if we continue without a major
> > >> > cleanup.
> > >> > > This makes it increasingly difficult to implement new features
> > >> > > effectively
> > >> > > while maintaining backward compatibility. A major release allows
> us
> > to
> > >> > > address these issues head-on, reducing technical debt and paving
> the
> > >> > way
> > >> > > for a more robust platform.
> > >> > > 3. *Managing Breaking Changes:* Let’s take the example of
> > restricting
> > >> > > database access from workers. It’s a necessary move for better
> > >> > security
> > >> > > and
> > >> > > also potentially scalability reasons (reduces DB load). Many users
> > >> > have
> > >> > > workflows that interact with the DB, either by using raw sql or by
> > >> > > leveraging a session object. We could implement this feature in
> > >> > Airflow
> > >> > > 2
> > >> > > and avoid breaking existing workflows by continuing to have the
> old
> > >> > > standard mode as default - much of the work is already done - but
> > that
> > >> > > would mean supporting both the new secure mode and the old
> standard
> > >> > mode
> > >> > > indefinitely and design new features with the assumption that most
> > >> > will
> > >> > > continue using the old standard mode. With Airflow 3, we can make
> > >> > secure
> > >> > > mode the default or even the only option, simplifying
> implementation
> > >> > and
> > >> > > future development. This is just one example where it is feasible
> to
> > >> > > implement in Airflow 2, but is better if we release it under the
> > >> > > context of
> > >> > > Airflow 3.
> > >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up
> > >> > > possibilities for handling workflows beyond batch processing.
> > Features
> > >> > > like
> > >> > > real-time DAG execution through API and multi-language task
> support
> > >> > are
> > >> > > big
> > >> > > steps forward, significantly expanding Airflow’s utility.
> > >> > >
> > >> > >
> > >> > > While integrating these updates into Airflow 2 might look less
> > >> disruptive
> > >> > > initially, the scale and nature of the required changes really
> > support
> > >> a
> > >> > > move to Airflow 3. It’s not just about adding new features; it’s
> > about
> > >> > > setting up Airflow so that it continues to remain relevant for the
> > next
> > >> > ten
> > >> > > years.
> > >> > >
> > >> > > Constance
> > >> > >
> > >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <ash@apache.org
> > >> <ma...@apache.org> <mailto:ash@apache.org <mailto:ash@apache.org
> > >>>
> > >> wrote:
> > >> > >
> > >> > > > There's a lot of technical debt hiding in Airflow, especially
> the
> > >> > > > scheduler that makes it harder and harder to efficiently add new
> > >> > > features.
> > >> > > >
> > >> > > > At some point, very soon, we are going to have to remove some
> very
> > >> > > > infrequently used back compat shims that negatively affect
> > >> performance.
> > >> > > > Without doing that the pace at which we can realistically add
> > some of
> > >> > the
> > >> > > > more exciting features tends towards zero. Developer speed of
> > >> > > contributors
> > >> > > > is a factor here too!
> > >> > > >
> > >> > > > So while we are still using SemVer, that necessitates v3.
> > >> > > >
> > >> > > > Ash
> > >> > > >
> > >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <
> > michalmodras@google.com
> > >> <ma...@google.com> <mailto:michalmodras@google.com
> > <mailto:
> > >> michalmodras@google.com>>
> > >> > > .INVALID>
> > >> > > > wrote:
> > >> > > > >+1 to Jens's & Bolke's points here and in the doc
> > >> > > > >
> > >> > > > >I agree we should work on clarifying the directions we would
> like
> > >> > > Airflow
> > >> > > > >to go. Introducing a new major Airflow version is a massive
> > overhead
> > >> > for
> > >> > > > >users, who would need to plan for migrations, onboarding the
> new
> > >> > Airflow
> > >> > > > >(with a slightly different architecture), etc., and effectively
> > >> > Airflow
> > >> > > 2
> > >> > > > >would live in parallel for a long time.
> > >> > > > >
> > >> > > > >Personally, I think most of the points in Kaxil's/Vikram's doc
> > are
> > >> > > > valuable
> > >> > > > >projects of their own, and I could imagine all of them being
> > >> delivered
> > >> > > as
> > >> > > > >separate AIPs within Airflow 2 (surely new minor versions of
> > Airflow
> > >> > > 2). I
> > >> > > > >am not sure if the scope of changes and the goal we want to
> > achieve
> > >> is
> > >> > > a)
> > >> > > > >clear enough b) broad enough to call for a new major version.
> > >> > > > >
> > >> > > > >Best,
> > >> > > > >Michal
> > >> > > > >
> > >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens
> (XC-AS/EAE-ADA-T)
> > >> > > > ><Jens.Scheffler@de.bosch.com.inva <mailto:
> > >> Jens.Scheffler@de.bosch.com.inva> <mailto:
> > Jens.Scheffler@de.bosch.com.inva
> > >> <ma...@de.bosch.com.inva>>lid> wrote:
> > >> > > > >
> > >> > > > >> Thanks for the document write-up, Kaxil. I assume this is
> > mostly a
> > >> > > > vision
> > >> > > > >> statement.
> > >> > > > >>
> > >> > > > >> Looking forward for a larger addendum where we can collect
> > things
> > >> > that
> > >> > > > we
> > >> > > > >> all can vote and agree on as targets.
> > >> > > > >>
> > >> > > > >> As I started earlier with a confluence page and it seems this
> > is
> > >> not
> > >> > > > >> accessible to all, shall we convert this to a Google Doc for
> > >> better
> > >> > > > >> collaboration and item collection?
> > >> > > > >>
> > >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> <
> > >> https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&gt;> <
> > >> https://aka.ms/o0ukef&amp;gt;&gt;>
> > >> > > > >> ________________________________
> > >> > > > >> From: Vikram Koka <vikram@astronomer.io.INVA <mailto:
> > >> vikram@astronomer.io.INVA> <mailto:vikram@astronomer.io.INVA <mailto:
> > >> vikram@astronomer.io.INVA>>LID>
> > >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> > >> > > > >> To: dev@airflow.apache.org <ma...@airflow.apache.org>
> > >> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>> <
> > >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> > >> dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow
> > 2)
> > >> vs
> > >> > > > >> strategic (Airflow 3) approach
> > >> > > > >>
> > >> > > > >> Thank you for your feedback, Bolke and Andrey!
> > >> > > > >>
> > >> > > > >> Bolke,
> > >> > > > >> I have replied to some of your comments in the doc.
> > >> > > > >> I will provide a detailed write up on the "Interactive DAG
> run"
> > >> (or
> > >> > > > >> synchronous DAG run) capability, which has generated some
> early
> > >> > > > questions.
> > >> > > > >> I had intended to get an AIP published for that as a
> follow-up,
> > >> but
> > >> > I
> > >> > > > >> believe that a simpler write up would be useful ahead of the
> > AIP.
> > >> > > > >>
> > >> > > > >> Andrey,
> > >> > > > >> You raise an interesting point.
> > >> > > > >>
> > >> > > > >> As part of the Airflow 2.0 release, we as a community had
> > decided
> > >> to
> > >> > > > >> strictly adhere to Semver as detailed in the document you
> > >> > referenced.
> > >> > > We
> > >> > > > >> also consciously split out the "Core Airflow" releases from
> the
> > >> > > > "Provider"
> > >> > > > >> releases at that time. We had a clear expectation then for
> the
> > >> > cadence
> > >> > > > of
> > >> > > > >> both minor and patch releases, which we have generally
> adhered
> > to
> > >> > > since
> > >> > > > >> then.
> > >> > > > >>
> > >> > > > >> Personally, I am more concerned about our Provider releases
> > right
> > >> > now,
> > >> > > > as
> > >> > > > >> compared to the cadence of our major releases. I believe that
> > one
> > >> of
> > >> > > the
> > >> > > > >> proposed changes in the Airflow 3 document i.e. the clear
> > >> separation
> > >> > > for
> > >> > > > >> Task Execution will help here, but more may be needed.
> > >> > > > >>
> > >> > > > >> Definitely interested in more feedback on this as well.
> > >> > > > >>
> > >> > > > >> Vikram
> > >> > > > >>
> > >> > > > >>
> > >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> > >> > > andrey.anshin@taragol.is <ma...@taragol.is>
> <mailto:
> > >> andrey.anshin@taragol.is <ma...@taragol.is>>
> > >> > > > >
> > >> > > > >> wrote:
> > >> > > > >>
> > >> > > > >> > I would like to propose to change (at least discuss)
> release
> > >> > policy
> > >> > > > >> around
> > >> > > > >> > the Major version of Airflow.
> > >> > > > >> >
> > >> > > > >> > Right now it is described as "These releases do not happen
> > with
> > >> > any
> > >> > > > >> regular
> > >> > > > >> > interval or on any predictable schedule." :
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> > >> <
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> > >
> > >> <
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> > >
> > >> <
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt
> > >> ;>
> > >> > > > >> <
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > >> <
> > >>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > >
> > >> <
> > >>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > >
> > >> <
> > >>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt
> > >> ;>
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > So maybe it is time to make it schedulable, e.g. one per
> two
> > >> years
> > >> > > or
> > >> > > > so.
> > >> > > > >> > This one could help us to avoid such a discussion in the
> > future,
> > >> > > like
> > >> > > > "We
> > >> > > > >> > don't know when Airflow 4 is coming.". At the moment when
> the
> > >> new
> > >> > > > major
> > >> > > > >> > version will be released new features wouldn't be added in
> > the
> > >> old
> > >> > > > major
> > >> > > > >> > version, however we would support bug / security for a
> while,
> > >> > e.g. 1
> > >> > > > year
> > >> > > > >> > for bug fixes, 3 years for security fixes with a total 5
> year
> > >> > > > lifecycle
> > >> > > > >> per
> > >> > > > >> > a major version. These just are approximate time periods
> for
> > a
> > >> > > > definition
> > >> > > > >> > of current period, bugfix period and security fix period.
> > >> > > > >> >
> > >> > > > >> > In contributors' perspective it helps with dropping the
> > >> deprecated
> > >> > > > stuff
> > >> > > > >> > which resolves some old problem: we have to support
> > everything
> > >> > > > including
> > >> > > > >> > deprecated stuff and without schedulable lifecycle for the
> > >> > > deprecated
> > >> > > > >> stuff
> > >> > > > >> > it could be showstopper for the new feature, because
> > sometimes
> > >> it
> > >> > > > hard to
> > >> > > > >> > support two different approaches for long period of time
> > with no
> > >> > > hope
> > >> > > > >> that
> > >> > > > >> > it will happen soon. For some fundamental stuff which do
> not
> > >> > > require a
> > >> > > > >> lot
> > >> > > > >> > things time to support we could postponed removal for next
> > after
> > >> > the
> > >> > > > next
> > >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in
> > Airflow 5
> > >> > > > >> >
> > >> > > > >> > In the user perspective, they have at least bug fix support
> > for
> > >> a
> > >> > > > while,
> > >> > > > >> if
> > >> > > > >> > someone want to use legacy version it their choice, however
> > no
> > >> new
> > >> > > > >> > features, no new version of providers (after one year)
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > ----
> > >> > > > >> > Best Wishes
> > >> > > > >> > *Andrey Anshin*
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <
> > bdbruin@gmail.com
> > >> <ma...@gmail.com> <mailto:bdbruin@gmail.com <mailto:
> > >> bdbruin@gmail.com>>>
> > >> > > > wrote:
> > >> > > > >> >
> > >> > > > >> > > I have left several comments :-). And on interactive dag
> > runs
> > >> > even
> > >> > > > >> after
> > >> > > > >> > > the explanation of Vikram I still don't have a clue what
> we
> > >> want
> > >> > > to
> > >> > > > >> > > accomplish there :-P.
> > >> > > > >> > >
> > >> > > > >> > > I would like to see a mantra or team for Airflow 3. That
> > helps
> > >> > > > nudging
> > >> > > > >> > > people in the same direction. Suggestions in the
> comments.
> > >> > > > >> > >
> > >> > > > >> > > Bolke
> > >> > > > >> > > Sent from my iPhone
> > >> > > > >> > >
> > >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> > >> > > <vikram@astronomer.io.inva <ma...@astronomer.io.inva>
> > <mailto:
> > >> vikram@astronomer.io.inva <ma...@astronomer.io.inva>>lid
> > >> > > > >
> > >> > > > >> > > wrote:
> > >> > > > >> > > >
> > >> > > > >> > > > Good point Jed.
> > >> > > > >> > > > I responded back to your comment in the doc as well and
> > very
> > >> > > open
> > >> > > > to
> > >> > > > >> > > > changing the term in the doc.
> > >> > > > >> > > >
> > >> > > > >> > > > Used the term "interactive DAG run" as the ability to
> > invoke
> > >> > or
> > >> > > > >> > trigger a
> > >> > > > >> > > > DAG run through the API, with the expectation of
> getting
> > >> back
> > >> > a
> > >> > > > >> result
> > >> > > > >> > > > immediately. An alternate term could be a "synchronous
> > DAG
> > >> > run".
> > >> > > > >> > > >
> > >> > > > >> > > > Regardless, this is a significant change so a good term
> > to
> > >> > > > indicate
> > >> > > > >> the
> > >> > > > >> > > > expansion from "batch runs only" is warranted. Very
> open
> > to
> > >> > > > different
> > >> > > > >> > > terms
> > >> > > > >> > > > here.
> > >> > > > >> > > >
> > >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > >> > > > >> > jedcunningham@apache.org <ma...@apache.org>
> > >> <mailto:jedcunningham@apache.org <ma...@apache.org>>
> > >> > > > >> > > >
> > >> > > > >> > > >> wrote:
> > >> > > > >> > > >>
> > >> > > > >> > > >> Very exciting! Looks like we will have a busy period
> of
> > >> time
> > >> > > > ahead
> > >> > > > >> of
> > >> > > > >> > > us.
> > >> > > > >> > > >> Overall I like the plan so far, especially using this
> > >> year's
> > >> > > > Airflow
> > >> > > > >> > > Summit
> > >> > > > >> > > >> as an opportunity to announce and gather feedback, and
> > the
> > >> > 2025
> > >> > > > >> > version
> > >> > > > >> > > to
> > >> > > > >> > > >> pitch upgrading.
> > >> > > > >> > > >>
> > >> > > > >> > > >> I left a comment in the doc, but we might want to
> > iterate
> > >> on
> > >> > > the
> > >> > > > >> > > >> terminology we use for high priority or "synchronous"
> > DAG
> > >> > runs
> > >> > > to
> > >> > > > >> > serve
> > >> > > > >> > > LLM
> > >> > > > >> > > >> responses - I find "interactive DAG runs" a bit
> > confusing.
> > >> > > > >> > > >>
> > >> > > > >> > >
> > >> > > > >> > >
> > >> > > >
> > ---------------------------------------------------------------------
> > >> > > > >> > > To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org
> > >> <ma...@airflow.apache.org> <mailto:
> > >> dev-unsubscribe@airflow.apache.org <mailto:
> > >> dev-unsubscribe@airflow.apache.org>>
> > >> > > > >> > > For additional commands, e-mail:
> > dev-help@airflow.apache.org
> > >> <ma...@airflow.apache.org> <mailto:
> > dev-help@airflow.apache.org
> > >> <ma...@airflow.apache.org>>
> > >> > > > >> > >
> > >> > > > >> > >
> > >> > > > >> >
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Amogh Desai <am...@gmail.com>]

Thanks for initiating this discussion @Kaxil Naik <ka...@astronomer.io>!


Read through the document and the emails here and I like the direction in
which
we are proceeding.

I also agree that removing code is almost as important as adding code. We
need to have a clear checklist of what we want to
remove and get a vote for the same from the community.

I also see that we have multiple ways of doing a thing, which is not needed
(serialising for example) and removing
which can lead to simpler interfaces for users, and contributors too.

I also find the *secret masker* overly complex personally, why can't it be
simpler :)

However, I have a small concern with the proportion of the users that would
like to migrate to an early version of
Airflow 3 if we do not support various deployment modes. If we dont get
enough initial feedback, we do not know how the new
major is doing, no?

Thanks & Regards,
Amogh Desai


On Thu, May 9, 2024 at 1:03 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> Strong agree - I think one of the things I regret most about Airflow 2 was
> that we didn't remove enough. It probably eased the upgrade process for
> users though.
>
> > but I was hoping one day when we remove providers
> > from main development, I can do it personally and beat Ash to it.
>
> Grrr ;l
>
> The other thing I want to look at (but don't have concrete ideas of yet)
> is some of the lesser used dag/task concurrency and dependency features -
> if we can remove a few things and get a faster scheduler than it's worth
> having a discussion over.
>
> On 9 May 2024 06:31:38 BST, Jarek Potiuk <ja...@potiuk.com> wrote:
> >I hope more voices will be coming soon as well :).
> >
> >Constance - few words of explanation, for my "remove" things. I do not say
> >we "have to" remove those things or even that we "should".
> >
> >My main motivation for mentioning the list is that we often forget that
> >when we have a new major version of software - removing stuff is as
> >important as adding. So I think when it comes to final decisions about
> >Airflow 3 we should be well aware about both: what we add and what we
> >remove. So my main point is - "removal" of things should be as important a
> >part in our future discussions as "adding", and I'd say with 10 years of
> >Airflow history, it's more important to remove things than to add them.
> >
> >Side comment: Ash for one mentioned at multiple occasions how proud he is
> >that his contributions to Airflow are net-negative - he removed more code
> >than added. Unfortunately it's not shown any more here
> >https://github.com/apache/airflow/graphs/contributors - because we have
> >more than 10.000 commits, but I was hoping one day when we remove
> providers
> >from main development, I can do it personally and beat Ash to it.
> >
> >Just wanted to stress a few things on why I think "discussing potential
> >removals" is important:
> >
> >* I think it is super important to deliver Airflow 3 really fast. It took
> >
> >1.5 year to release Airflow 2 and it was far too long and keeping Airflow
> 1
> >and 2 in parallel was a huge pain and drain and it slowed us down
> >enormously. We should absolutely limit the time when we actively develop
> >both Airflow 2 features and Airflow 3 features.
> >* I think most of the current deprecations of the 100+ we have does not
> >slow us down AT ALL. Removal of those is mostly cosmetic change, a little
> >bit clutter to remove but they have little-to-no impact on actual speed of
> >development, it's just an old code that keeps on being around
> >* on the other hand - some of the things I mentioned ARE slowing us down A
> >LOT and will continue to do so until we remove them. For example, I
> believe
> >"postgresql + mysql + sqlite complete versioning solution for all the
> >possible variants of storage" will be quite a bit more complex and testing
> >will take far more time than if we drop mysql and choose a single storage
> >approach for Airflow 3.0. I have no hard data to back it up, but my gut
> >feeling is that it can take at least twice as long to get it out in the
> >hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for
> >all the options we have there.
> >* the telemetry will be cool - but even if we add it for 2.10, the first
> >time meaningful data will be available is maybe 2 years from now when we
> do
> >Airflow 4. For now we can completely forget about it because it's only
> >going to show us some early adopters of Airflow 2.10. But we have surveys
> +
> >we can collect data from Astronomer, Google. Amazon for some usage and
> >identify who will be early adopters of Airflow 3 and target them first.
> >* the fact that we will drop something for Airflow 3.0, does not mean that
> >we have to drop it "forever". We can safely assume that Airflow 3.0 will
> be
> >only used by early adopters, and many of our users will wait for 3.1, 3.2
> >or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those "slower
> >moving users" from the 3.0. We can re-add things in 3.1 building on
> >foundations of 3.0 after it is in the hands of those early adopters. I
> >personally think we should prioritize speed of delivery of 3.0 over
> >"complete support of every deployment option of Airflow 2" - to allow
> >**some** of our early adopter users to migrate quickly, and add what's
> >missing for those who would like to move later in later versions.
> >* I think - this is the most important "product management" decision here
> -
> >based on the data we have available. Which users do we target for Airflow
> >3.0, and what we can drop to deliver it faster (and possibly re-add more
> >stuff for 3.1+). I think it's crucial to the success of the Airflow 3.0
> >initiative and the most important decision from the Product management
> side
> >that will impact the timeline of Airflow 3.0. While a lot depends on how
> >much time and effort individuals will spend on implementing things, the
> >decision of what we drop will impact everyone's speed and overall delivery
> >of Airflow 3.0. It's a decision we should not take lightly.
> >
> >So my main point is: let's add "what we remove" as a very important point
> >in the product discussions we are going to have.
> >
> >J.
> >
> >
> >
> >
> >On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar
> ><rb...@amazon.com.invalid> wrote:
> >
> >> Like Constance and many others, there was time needed to process this
> >> fantastical summary and approach to Airflow 3. __
> >> A lot of the points raised by Jarek make sense, ie being prescriptive on
> >> what features we want to be there on Day 1 vs launched on 3.x - this
> does
> >> allow us to move quickly for our users.
> >> In line with Constance's concern, we need to be mindful of what we
> decide
> >> to keep and what we are willing to cut - which could be a painstaking
> >> process that could offset any gains of trying to be faster. I also
> believe
> >> that bringing all these new amazing features on Airflow 3 will peak the
> >> interest of early adopters and eventually get others interested in
> >> migration. However, I believe this migration will be a slow process and
> >> will present a gap in certain functionalities that users may want before
> >> entertaining any move to Airflow 3. There are still a lot of folks using
> >> v1.10 today. There were several tactical initiatives in the past few
> months
> >> with intent on bringing new functionality, ie Multi-team, to Airflow 2.x
> >> and I feel that while these efforts are not wasted, there should still
> be
> >> an option to continue improving Airflow 2 to avoid alienating our users
> on
> >> the basis of a future promise in Airflow 3, that may not be easy to
> migrate
> >> towards.
> >>
> >> -- Rajesh
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> dev@airflow.apache.org <ma...@airflow.apache.org> <
> >> dev@airflow.apache.org <ma...@airflow.apache.org>>
> >>
> >>
> >> On 2024-05-07, 12:26 PM, "Constance Martineau"
> >> <constance@astronomer.io.INVA <ma...@astronomer.io.INVA>
> >> <mailto:constance@astronomer.io.INVA <mailto:
> constance@astronomer.io.INVA>>LID>
> >> wrote:
> >>
> >>
> >>
> >>
> >> CAUTION: This email originated from outside of the organization. Do not
> >> click links or open attachments unless you can confirm the sender and
> know
> >> the content is safe.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe.
> >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> >> le contenu ne présente aucun risque.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Thank you Jarek for the detailed input. I've taken some time to digest
> your
> >> points before responding.
> >>
> >>
> >>
> >>
> >> You've outlined a bold vision for Airflow 3, and I agree that being
> >> decisive about the features and architectures will be the key to
> success.
> >> However, before we make final decisions on what features to cut or
> retain,
> >> it would be beneficial to have a more comprehensive understanding of how
> >> the current features are utilized by the open-source community.
> >>
> >>
> >>
> >>
> >> @Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io> <mailto:
> >> kaxil@astronomer.io <ma...@astronomer.io>>> recently initiated a
> >> discussion on
> >> collecting telemetry from open-source deployments:
> >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m <
> >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> >> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;>.
> >> This data
> >> could be critical in ensuring our decisions are well-informed and
> reflect
> >> the real-world usage patterns of our users, not just those from managed
> >> environments like Astro, MWAA or GCC.
> >>
> >>
> >>
> >>
> >> It's essential that we challenge our assumptions and base our decisions
> on
> >> a holistic view of feature usage. Identifying potential cuts is a
> critical
> >> step, but let's ensure our strategy aligns with the needs and
> preferences
> >> of the broader Airflow community.
> >>
> >>
> >>
> >>
> >> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <jarek@potiuk.com <mailto:
> >> jarek@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> >> wrote:
> >>
> >>
> >>
> >>
> >> > I am currently on sick leave, and still recovering - hoping to be
> able to
> >> > travel next week to the US as planned, so I just wanted to break out
> of
> >> it
> >> > to make one comment here.
> >> >
> >> > I got a clearer head now a bit with medications hopefully working. I
> am
> >> > still taking it that should help me to get over the current state,
> and I
> >> > wanted to take a look at this discussion unraveling first. Over last
> week
> >> > I disconnected from "day-to-day" Airflow and put some thoughts (as
> much
> >> as
> >> > I could in my current state) on it. The whole subject of this thread
> was
> >> > started from that - how the current discussions on AIP-67 and others
> >> change
> >> > if we consider Airflow 3 is "starting".
> >> >
> >> > The price for back-compat is speed of development and quality. More
> >> > combinations to test, more unexpected issues uncovered, necessity to
> keep
> >> > parallel paths (old/new) while adding new features. All what Constance
> >> > wrote about and what Ash explained. We already started to trip over
> our
> >> own
> >> > feet mutliple times in a few last releases. Have we tested all
> >> combinations
> >> > of deployment in Airflow 2.8 and 2.9 - not really, I think we already
> see
> >> > that in a number of "combos" of features things are not working in as
> >> > stable a way as they did before.
> >> >
> >> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a
> long
> >> > time (or even move out) as they will not want to move to Airflow 3. A
> lot
> >> > of the work implemented in AIP-44 and design of AIP-67 was done around
> >> > back-compatibility. but yes -
> >> > it would have been way easier if designed anew without
> back-compatibility
> >> > in mind. And if we implement it and release it in Airflow 2 it will
> make
> >> > new Airflow feature development even harder. That's why I wanted to
> treat
> >> > it as "tactical" solution - hoping that in Airflow 3 we can make it
> >> > "properly" - and that's why I started the discussion here when I
> sensed
> >> > that we are "close" to Airflow 3 discussion, because I wanted to see
> what
> >> > options we have there. This is why I have not yet concluded voting on
> >> > AIP-67 waiting for the result of this discussion here.
> >> >
> >> > But if we are ready to go for Airflow.3 then I'd say there are two
> >> > important things that should be part of the vision.
> >> >
> >> > 1) *We should be far more opinionated and have far fewer options of
> >> > running things in Airflow 3*. Even an order of magnitude more
> >> opinionated.
> >> > Make choices, stick to it, perfect those opinionated choices to suit
> >> 80/20
> >> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not
> fitting
> >> > the 20% that might choose to stay at Airflow 2. We can choose now
> which
> >> > ~20% of cases we do not want to handle deliberately. And we should be
> >> very,
> >> > very strict about it. Default should be "no choice". This will
> radically
> >> > simplify deployment and should make it easier to simplify Airflow
> >> > development and DAG authoring experience because we will have less
> cases
> >> to
> >> > support. Even if we plan to add more options in the future, the first
> >> > version of Airflow 3 should support one deployment approach only.
> This is
> >> > the only way we can deliver it fast. And we should be very bold there.
> >> > Choose one option and go for it in pretty much every place we have
> >> choices
> >> > now. We should Aim for Airflow 3.0 to support only a subset of current
> >> > users - but those who are most likely to migrate first and those with
> the
> >> > biggest need for the new features. We can think 3.x to support more
> >> cases,
> >> > but 3.0 should be as opinionated as humanly possible.
> >> >
> >> > And this deployment option should be also something ALL our
> stakeholders
> >> > will feel OK with as a way forward in their offering.
> >> >
> >> > My candidates (and yes, some are bold):
> >> >
> >> > * *Drop MySQL*. If we have a single thing that makes us avoid our
> schema
> >> > and DB migration - this is the case. Let's choose Postgres 15+ and use
> >> some
> >> > of the great features there. This will also enable much faster async
> SQL
> >> > implementation and a number of other optimisations - not to mention
> >> cutting
> >> > every single change in development and testing time by literally half.
> >> And
> >> > we should not look back to adding MySQL.
> >> > * *Drop Celery/Sequential Executor* and start with Local + K8S only
> (and
> >> > AWS/Google others can continue developing theirs of course in parallel
> >> and
> >> > continue Hybrid executor work). Later - we figure out a better
> solution
> >> to
> >> > support "small" tasks using some new K8S features and possibly non-k8s
> >> > solutions (Ray-based?)
> >> > * *Cut Connection and Variable Management from DB/UI*. Leave only
> Secrets
> >> > Management. Later when we have a 100% extensible React UI, we can add
> a
> >> > "local DB secrets manager" add-on
> >> > * *Choose a single way for DAG storage that will support versioning
> from
> >> > day one*. Bear in mind we can add others later. Bolke's idea of using
> >> > FSspec is an interesting one, we should see if it is feasible.
> >> > * *Drop FAB completely (including custom plugins) and invest in
> >> > implementing Auth Manager based on a dedicated, external solution*
> >> > (KeyCloak
> >> > that we've discussed before as a likely candidate)
> >> > * *Leave Providers with Airflow 2 and add tests to make sure they are
> >> > Airflow 3 future-compatible *- develop a way where we continue
> >> development
> >> > and contributions for Providers with Airflow 2 and add complete tests
> to
> >> > run them with Airflow 3. This way we can continue developing Provider
> >> > features independently, and make them work for Airflow 2 (and continue
> >> > adding features for Airflow 2 users alongside Airflow 2 bugfixes),
> while
> >> > also gradually fix any Airflow3 incompatibilities and instead of
> >> > "back-compatibility" tests make provider "forward-compatibility"
> tests so
> >> > that future Providers are tested and work on Airflow 3. Also it will
> make
> >> > it easiest to continue Airflow 2 (bugfixes) + Providers tested without
> >> > investing in changing the current CI / test harness.
> >> > * *Simplify Test Harness for Airflow 3 from the start *- without
> >> providers
> >> > and 790+ dependencies, we could vastly simplify Airflow3 testing
> >> (basically
> >> > make CI jobs from scratch) using mostly standard Python tooling
> (while we
> >> > can continue making use of the current test harness for Airflow 2 +
> >> > Providers and extend it with Airflow 3 future-compatibility tests).
> That
> >> > means Breeze would be only staying in Airflow 2 + Providers repo as we
> >> > should be able to achieve most of what we have there with local venv/
> >> > tooling (especially with uv as underlying tooling).
> >> >
> >> > 2) *I think we only add very few new "important" features. *Absolute
> >> > minimum to make Airflow 3 appealing and add them only in Airflow 3:
> >> > versioning, multi-team, pluggable UI should only be Airflow 3 - it
> makes
> >> no
> >> > sense to invest into Airflow 2 if we already know Airflow 3 is coming
> -
> >> > that generally triples effort needed to get them out. We should drop
> new
> >> > features development in Airflow 2. This will give users incentive to
> move
> >> > to 3 if the new features will be worth it. Even paying
> >> > compatibility/migration price.
> >> >
> >> > Versionig, for example: I believe if we decide to go only with
> Airflow 3
> >> > and cut some of the above (Postgres only, Single versioning DAG
> storage)
> >> we
> >> > can make bolder decisions in versioning and support simpler models
> from
> >> the
> >> > get go (and deliver it faster). And we should add only a few - but
> >> > important - features that our users clearly asked for and focus on
> >> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or
> >> 2.11).
> >> > Similarly - multi-team can be simplified if we cut things from the
> list
> >> > above and have Task isolation as first-class citizens in Airflow (and
> the
> >> > only option).
> >> >
> >> > My candidates very much concur with the list shared by Kaxil in the
> doc +
> >> > I'd add multi-team (but simplified thanks to the cuts). But I also
> here
> >> > would mostly revert to Astronomer, Google. AWS team to define
> >> collectively
> >> > what is the absolute minimum set of features that would get the
> "target"
> >> > part of their customers happy. And ONLY do that.
> >> >
> >> > So in short - I think the big part of our discussion should be what we
> >> are
> >> > ready to drop when we start airflow 3 and be very bold. Once we know
> we
> >> > should figure out the absolute minimum of things that we can add that
> >> will
> >> > benefit a significant part of our users (and make use of increased
> speed
> >> > because we dropped things).
> >> >
> >> > J.
> >> >
> >> >
> >> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau
> >> > <constance@astronomer.io.inva <ma...@astronomer.io.inva>
> >> <mailto:constance@astronomer.io.inva <mailto:
> constance@astronomer.io.inva>>lid>
> >> wrote:
> >> >
> >> > > Hi Michal,
> >> > >
> >> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate
> your
> >> > > concerns about the migration overhead for our users with a major new
> >> > > version and see the appeal in your suggestion to integrate many of
> the
> >> > > proposed changes into Airflow 2 through separate AIPs. It’s a valid
> >> point
> >> > > and certainly aligns with the value of making incremental
> improvements.
> >> > >
> >> > > However, after looking closely at the enhancements outlined for
> Airflow
> >> > 3,
> >> > > I'm convinced they warrant a new major release. Here’s why:
> >> > >
> >> > > 1. *Core Architectural Changes:* We’re looking at foundational
> changes
> >> > > with Airflow 3—like redefining task priorities, separating task
> >> > > definition
> >> > > and task execution, and new AIPs like DAG versioning. remote
> execution
> >> > > and restricting database access from workers. These aren’t just
> >> > > incremental
> >> > > improvements but major shifts that will set the stage for the next
> >> > > decade
> >> > > of Airflow’s architecture. Grouping these changes into a major
> release
> >> > > will
> >> > > help us make these transitions more cleanly and with fewer
> constraints
> >> > > from
> >> > > past decisions.
> >> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140
> >> > deprecated
> >> > > issues, and this will only grow if we continue without a major
> >> > cleanup.
> >> > > This makes it increasingly difficult to implement new features
> >> > > effectively
> >> > > while maintaining backward compatibility. A major release allows us
> to
> >> > > address these issues head-on, reducing technical debt and paving the
> >> > way
> >> > > for a more robust platform.
> >> > > 3. *Managing Breaking Changes:* Let’s take the example of
> restricting
> >> > > database access from workers. It’s a necessary move for better
> >> > security
> >> > > and
> >> > > also potentially scalability reasons (reduces DB load). Many users
> >> > have
> >> > > workflows that interact with the DB, either by using raw sql or by
> >> > > leveraging a session object. We could implement this feature in
> >> > Airflow
> >> > > 2
> >> > > and avoid breaking existing workflows by continuing to have the old
> >> > > standard mode as default - much of the work is already done - but
> that
> >> > > would mean supporting both the new secure mode and the old standard
> >> > mode
> >> > > indefinitely and design new features with the assumption that most
> >> > will
> >> > > continue using the old standard mode. With Airflow 3, we can make
> >> > secure
> >> > > mode the default or even the only option, simplifying implementation
> >> > and
> >> > > future development. This is just one example where it is feasible to
> >> > > implement in Airflow 2, but is better if we release it under the
> >> > > context of
> >> > > Airflow 3.
> >> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up
> >> > > possibilities for handling workflows beyond batch processing.
> Features
> >> > > like
> >> > > real-time DAG execution through API and multi-language task support
> >> > are
> >> > > big
> >> > > steps forward, significantly expanding Airflow’s utility.
> >> > >
> >> > >
> >> > > While integrating these updates into Airflow 2 might look less
> >> disruptive
> >> > > initially, the scale and nature of the required changes really
> support
> >> a
> >> > > move to Airflow 3. It’s not just about adding new features; it’s
> about
> >> > > setting up Airflow so that it continues to remain relevant for the
> next
> >> > ten
> >> > > years.
> >> > >
> >> > > Constance
> >> > >
> >> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <ash@apache.org
> >> <ma...@apache.org> <mailto:ash@apache.org <mailto:ash@apache.org
> >>>
> >> wrote:
> >> > >
> >> > > > There's a lot of technical debt hiding in Airflow, especially the
> >> > > > scheduler that makes it harder and harder to efficiently add new
> >> > > features.
> >> > > >
> >> > > > At some point, very soon, we are going to have to remove some very
> >> > > > infrequently used back compat shims that negatively affect
> >> performance.
> >> > > > Without doing that the pace at which we can realistically add
> some of
> >> > the
> >> > > > more exciting features tends towards zero. Developer speed of
> >> > > contributors
> >> > > > is a factor here too!
> >> > > >
> >> > > > So while we are still using SemVer, that necessitates v3.
> >> > > >
> >> > > > Ash
> >> > > >
> >> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <
> michalmodras@google.com
> >> <ma...@google.com> <mailto:michalmodras@google.com
> <mailto:
> >> michalmodras@google.com>>
> >> > > .INVALID>
> >> > > > wrote:
> >> > > > >+1 to Jens's & Bolke's points here and in the doc
> >> > > > >
> >> > > > >I agree we should work on clarifying the directions we would like
> >> > > Airflow
> >> > > > >to go. Introducing a new major Airflow version is a massive
> overhead
> >> > for
> >> > > > >users, who would need to plan for migrations, onboarding the new
> >> > Airflow
> >> > > > >(with a slightly different architecture), etc., and effectively
> >> > Airflow
> >> > > 2
> >> > > > >would live in parallel for a long time.
> >> > > > >
> >> > > > >Personally, I think most of the points in Kaxil's/Vikram's doc
> are
> >> > > > valuable
> >> > > > >projects of their own, and I could imagine all of them being
> >> delivered
> >> > > as
> >> > > > >separate AIPs within Airflow 2 (surely new minor versions of
> Airflow
> >> > > 2). I
> >> > > > >am not sure if the scope of changes and the goal we want to
> achieve
> >> is
> >> > > a)
> >> > > > >clear enough b) broad enough to call for a new major version.
> >> > > > >
> >> > > > >Best,
> >> > > > >Michal
> >> > > > >
> >> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> >> > > > ><Jens.Scheffler@de.bosch.com.inva <mailto:
> >> Jens.Scheffler@de.bosch.com.inva> <mailto:
> Jens.Scheffler@de.bosch.com.inva
> >> <ma...@de.bosch.com.inva>>lid> wrote:
> >> > > > >
> >> > > > >> Thanks for the document write-up, Kaxil. I assume this is
> mostly a
> >> > > > vision
> >> > > > >> statement.
> >> > > > >>
> >> > > > >> Looking forward for a larger addendum where we can collect
> things
> >> > that
> >> > > > we
> >> > > > >> all can vote and agree on as targets.
> >> > > > >>
> >> > > > >> As I started earlier with a confluence page and it seems this
> is
> >> not
> >> > > > >> accessible to all, shall we convert this to a Google Doc for
> >> better
> >> > > > >> collaboration and item collection?
> >> > > > >>
> >> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> <
> >> https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&gt;> <
> >> https://aka.ms/o0ukef&amp;gt;&gt;>
> >> > > > >> ________________________________
> >> > > > >> From: Vikram Koka <vikram@astronomer.io.INVA <mailto:
> >> vikram@astronomer.io.INVA> <mailto:vikram@astronomer.io.INVA <mailto:
> >> vikram@astronomer.io.INVA>>LID>
> >> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> >> > > > >> To: dev@airflow.apache.org <ma...@airflow.apache.org>
> >> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>> <
> >> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> >> dev@airflow.apache.org <ma...@airflow.apache.org>>>
> >> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow
> 2)
> >> vs
> >> > > > >> strategic (Airflow 3) approach
> >> > > > >>
> >> > > > >> Thank you for your feedback, Bolke and Andrey!
> >> > > > >>
> >> > > > >> Bolke,
> >> > > > >> I have replied to some of your comments in the doc.
> >> > > > >> I will provide a detailed write up on the "Interactive DAG run"
> >> (or
> >> > > > >> synchronous DAG run) capability, which has generated some early
> >> > > > questions.
> >> > > > >> I had intended to get an AIP published for that as a follow-up,
> >> but
> >> > I
> >> > > > >> believe that a simpler write up would be useful ahead of the
> AIP.
> >> > > > >>
> >> > > > >> Andrey,
> >> > > > >> You raise an interesting point.
> >> > > > >>
> >> > > > >> As part of the Airflow 2.0 release, we as a community had
> decided
> >> to
> >> > > > >> strictly adhere to Semver as detailed in the document you
> >> > referenced.
> >> > > We
> >> > > > >> also consciously split out the "Core Airflow" releases from the
> >> > > > "Provider"
> >> > > > >> releases at that time. We had a clear expectation then for the
> >> > cadence
> >> > > > of
> >> > > > >> both minor and patch releases, which we have generally adhered
> to
> >> > > since
> >> > > > >> then.
> >> > > > >>
> >> > > > >> Personally, I am more concerned about our Provider releases
> right
> >> > now,
> >> > > > as
> >> > > > >> compared to the cadence of our major releases. I believe that
> one
> >> of
> >> > > the
> >> > > > >> proposed changes in the Airflow 3 document i.e. the clear
> >> separation
> >> > > for
> >> > > > >> Task Execution will help here, but more may be needed.
> >> > > > >>
> >> > > > >> Definitely interested in more feedback on this as well.
> >> > > > >>
> >> > > > >> Vikram
> >> > > > >>
> >> > > > >>
> >> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> >> > > andrey.anshin@taragol.is <ma...@taragol.is> <mailto:
> >> andrey.anshin@taragol.is <ma...@taragol.is>>
> >> > > > >
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >> > I would like to propose to change (at least discuss) release
> >> > policy
> >> > > > >> around
> >> > > > >> > the Major version of Airflow.
> >> > > > >> >
> >> > > > >> > Right now it is described as "These releases do not happen
> with
> >> > any
> >> > > > >> regular
> >> > > > >> > interval or on any predictable schedule." :
> >> > > > >> >
> >> > > > >> >
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> >> <
> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> >
> >> <
> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0
> >
> >> <
> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt
> >> ;>
> >> > > > >> <
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> >> <
> >>
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> >
> >> <
> >>
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> >
> >> <
> >>
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt
> >> ;>
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > So maybe it is time to make it schedulable, e.g. one per two
> >> years
> >> > > or
> >> > > > so.
> >> > > > >> > This one could help us to avoid such a discussion in the
> future,
> >> > > like
> >> > > > "We
> >> > > > >> > don't know when Airflow 4 is coming.". At the moment when the
> >> new
> >> > > > major
> >> > > > >> > version will be released new features wouldn't be added in
> the
> >> old
> >> > > > major
> >> > > > >> > version, however we would support bug / security for a while,
> >> > e.g. 1
> >> > > > year
> >> > > > >> > for bug fixes, 3 years for security fixes with a total 5 year
> >> > > > lifecycle
> >> > > > >> per
> >> > > > >> > a major version. These just are approximate time periods for
> a
> >> > > > definition
> >> > > > >> > of current period, bugfix period and security fix period.
> >> > > > >> >
> >> > > > >> > In contributors' perspective it helps with dropping the
> >> deprecated
> >> > > > stuff
> >> > > > >> > which resolves some old problem: we have to support
> everything
> >> > > > including
> >> > > > >> > deprecated stuff and without schedulable lifecycle for the
> >> > > deprecated
> >> > > > >> stuff
> >> > > > >> > it could be showstopper for the new feature, because
> sometimes
> >> it
> >> > > > hard to
> >> > > > >> > support two different approaches for long period of time
> with no
> >> > > hope
> >> > > > >> that
> >> > > > >> > it will happen soon. For some fundamental stuff which do not
> >> > > require a
> >> > > > >> lot
> >> > > > >> > things time to support we could postponed removal for next
> after
> >> > the
> >> > > > next
> >> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in
> Airflow 5
> >> > > > >> >
> >> > > > >> > In the user perspective, they have at least bug fix support
> for
> >> a
> >> > > > while,
> >> > > > >> if
> >> > > > >> > someone want to use legacy version it their choice, however
> no
> >> new
> >> > > > >> > features, no new version of providers (after one year)
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > ----
> >> > > > >> > Best Wishes
> >> > > > >> > *Andrey Anshin*
> >> > > > >> >
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <
> bdbruin@gmail.com
> >> <ma...@gmail.com> <mailto:bdbruin@gmail.com <mailto:
> >> bdbruin@gmail.com>>>
> >> > > > wrote:
> >> > > > >> >
> >> > > > >> > > I have left several comments :-). And on interactive dag
> runs
> >> > even
> >> > > > >> after
> >> > > > >> > > the explanation of Vikram I still don't have a clue what we
> >> want
> >> > > to
> >> > > > >> > > accomplish there :-P.
> >> > > > >> > >
> >> > > > >> > > I would like to see a mantra or team for Airflow 3. That
> helps
> >> > > > nudging
> >> > > > >> > > people in the same direction. Suggestions in the comments.
> >> > > > >> > >
> >> > > > >> > > Bolke
> >> > > > >> > > Sent from my iPhone
> >> > > > >> > >
> >> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> >> > > <vikram@astronomer.io.inva <ma...@astronomer.io.inva>
> <mailto:
> >> vikram@astronomer.io.inva <ma...@astronomer.io.inva>>lid
> >> > > > >
> >> > > > >> > > wrote:
> >> > > > >> > > >
> >> > > > >> > > > Good point Jed.
> >> > > > >> > > > I responded back to your comment in the doc as well and
> very
> >> > > open
> >> > > > to
> >> > > > >> > > > changing the term in the doc.
> >> > > > >> > > >
> >> > > > >> > > > Used the term "interactive DAG run" as the ability to
> invoke
> >> > or
> >> > > > >> > trigger a
> >> > > > >> > > > DAG run through the API, with the expectation of getting
> >> back
> >> > a
> >> > > > >> result
> >> > > > >> > > > immediately. An alternate term could be a "synchronous
> DAG
> >> > run".
> >> > > > >> > > >
> >> > > > >> > > > Regardless, this is a significant change so a good term
> to
> >> > > > indicate
> >> > > > >> the
> >> > > > >> > > > expansion from "batch runs only" is warranted. Very open
> to
> >> > > > different
> >> > > > >> > > terms
> >> > > > >> > > > here.
> >> > > > >> > > >
> >> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> >> > > > >> > jedcunningham@apache.org <ma...@apache.org>
> >> <mailto:jedcunningham@apache.org <ma...@apache.org>>
> >> > > > >> > > >
> >> > > > >> > > >> wrote:
> >> > > > >> > > >>
> >> > > > >> > > >> Very exciting! Looks like we will have a busy period of
> >> time
> >> > > > ahead
> >> > > > >> of
> >> > > > >> > > us.
> >> > > > >> > > >> Overall I like the plan so far, especially using this
> >> year's
> >> > > > Airflow
> >> > > > >> > > Summit
> >> > > > >> > > >> as an opportunity to announce and gather feedback, and
> the
> >> > 2025
> >> > > > >> > version
> >> > > > >> > > to
> >> > > > >> > > >> pitch upgrading.
> >> > > > >> > > >>
> >> > > > >> > > >> I left a comment in the doc, but we might want to
> iterate
> >> on
> >> > > the
> >> > > > >> > > >> terminology we use for high priority or "synchronous"
> DAG
> >> > runs
> >> > > to
> >> > > > >> > serve
> >> > > > >> > > LLM
> >> > > > >> > > >> responses - I find "interactive DAG runs" a bit
> confusing.
> >> > > > >> > > >>
> >> > > > >> > >
> >> > > > >> > >
> >> > > >
> ---------------------------------------------------------------------
> >> > > > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> <ma...@airflow.apache.org> <mailto:
> >> dev-unsubscribe@airflow.apache.org <mailto:
> >> dev-unsubscribe@airflow.apache.org>>
> >> > > > >> > > For additional commands, e-mail:
> dev-help@airflow.apache.org
> >> <ma...@airflow.apache.org> <mailto:
> dev-help@airflow.apache.org
> >> <ma...@airflow.apache.org>>
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Ash Berlin-Taylor <as...@apache.org>]

Strong agree - I think one of the things I regret most about Airflow 2 was that we didn't remove enough. It probably eased the upgrade process for users though.

> but I was hoping one day when we remove providers
> from main development, I can do it personally and beat Ash to it.

Grrr ;l

The other thing I want to look at (but don't have concrete ideas of yet) is some of the lesser used dag/task concurrency and dependency features - if we can remove a few things and get a faster scheduler than it's worth having a discussion over.

On 9 May 2024 06:31:38 BST, Jarek Potiuk <ja...@potiuk.com> wrote:
>I hope more voices will be coming soon as well :).
>
>Constance - few words of explanation, for my "remove" things. I do not say
>we "have to" remove those things or even that we "should".
>
>My main motivation for mentioning the list is that we often forget that
>when we have a new major version of software - removing stuff is as
>important as adding. So I think when it comes to final decisions about
>Airflow 3 we should be well aware about both: what we add and what we
>remove. So my main point is - "removal" of things should be as important a
>part in our future discussions as "adding", and I'd say with 10 years of
>Airflow history, it's more important to remove things than to add them.
>
>Side comment: Ash for one mentioned at multiple occasions how proud he is
>that his contributions to Airflow are net-negative - he removed more code
>than added. Unfortunately it's not shown any more here
>https://github.com/apache/airflow/graphs/contributors - because we have
>more than 10.000 commits, but I was hoping one day when we remove providers
>from main development, I can do it personally and beat Ash to it.
>
>Just wanted to stress a few things on why I think "discussing potential
>removals" is important:
>
>* I think it is super important to deliver Airflow 3 really fast. It took >
>1.5 year to release Airflow 2 and it was far too long and keeping Airflow 1
>and 2 in parallel was a huge pain and drain and it slowed us down
>enormously. We should absolutely limit the time when we actively develop
>both Airflow 2 features and Airflow 3 features.
>* I think most of the current deprecations of the 100+ we have does not
>slow us down AT ALL. Removal of those is mostly cosmetic change, a little
>bit clutter to remove but they have little-to-no impact on actual speed of
>development, it's just an old code that keeps on being around
>* on the other hand - some of the things I mentioned ARE slowing us down A
>LOT and will continue to do so until we remove them. For example, I believe
>"postgresql + mysql + sqlite complete versioning solution for all the
>possible variants of storage" will be quite a bit more complex and testing
>will take far more time than if we drop mysql and choose a single storage
>approach for Airflow 3.0. I have no hard data to back it up, but my gut
>feeling is that it can take at least twice as long to get it out in the
>hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for
>all the options we have there.
>* the telemetry will be cool - but even if we add it for 2.10, the first
>time meaningful data will be available is maybe 2 years from now when we do
>Airflow 4. For now we can completely forget about it because it's only
>going to show us some early adopters of Airflow 2.10. But we have surveys +
>we can collect data from Astronomer, Google. Amazon for some usage and
>identify who will be early adopters of Airflow 3 and target them first.
>* the fact that we will drop something for Airflow 3.0, does not mean that
>we have to drop it "forever". We can safely assume that Airflow 3.0 will be
>only used by early adopters, and many of our users will wait for 3.1, 3.2
>or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those "slower
>moving users" from the 3.0. We can re-add things in 3.1 building on
>foundations of 3.0 after it is in the hands of those early adopters. I
>personally think we should prioritize speed of delivery of 3.0 over
>"complete support of every deployment option of Airflow 2" - to allow
>**some** of our early adopter users to migrate quickly, and add what's
>missing for those who would like to move later in later versions.
>* I think - this is the most important "product management" decision here -
>based on the data we have available. Which users do we target for Airflow
>3.0, and what we can drop to deliver it faster (and possibly re-add more
>stuff for 3.1+). I think it's crucial to the success of the Airflow 3.0
>initiative and the most important decision from the Product management side
>that will impact the timeline of Airflow 3.0. While a lot depends on how
>much time and effort individuals will spend on implementing things, the
>decision of what we drop will impact everyone's speed and overall delivery
>of Airflow 3.0. It's a decision we should not take lightly.
>
>So my main point is: let's add "what we remove" as a very important point
>in the product discussions we are going to have.
>
>J.
>
>
>
>
>On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar
><rb...@amazon.com.invalid> wrote:
>
>> Like Constance and many others, there was time needed to process this
>> fantastical summary and approach to Airflow 3. __
>> A lot of the points raised by Jarek make sense, ie being prescriptive on
>> what features we want to be there on Day 1 vs launched on 3.x - this does
>> allow us to move quickly for our users.
>> In line with Constance's concern, we need to be mindful of what we decide
>> to keep and what we are willing to cut - which could be a painstaking
>> process that could offset any gains of trying to be faster. I also believe
>> that bringing all these new amazing features on Airflow 3 will peak the
>> interest of early adopters and eventually get others interested in
>> migration. However, I believe this migration will be a slow process and
>> will present a gap in certain functionalities that users may want before
>> entertaining any move to Airflow 3. There are still a lot of folks using
>> v1.10 today. There were several tactical initiatives in the past few months
>> with intent on bringing new functionality, ie Multi-team, to Airflow 2.x
>> and I feel that while these efforts are not wasted, there should still be
>> an option to continue improving Airflow 2 to avoid alienating our users on
>> the basis of a future promise in Airflow 3, that may not be easy to migrate
>> towards.
>>
>> -- Rajesh
>>
>>
>>
>>
>>
>>
>>
>>
>> dev@airflow.apache.org <ma...@airflow.apache.org> <
>> dev@airflow.apache.org <ma...@airflow.apache.org>>
>>
>>
>> On 2024-05-07, 12:26 PM, "Constance Martineau"
>> <constance@astronomer.io.INVA <ma...@astronomer.io.INVA>
>> <mailto:constance@astronomer.io.INVA <ma...@astronomer.io.INVA>>LID>
>> wrote:
>>
>>
>>
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you can confirm the sender and know
>> the content is safe.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
>> le contenu ne présente aucun risque.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Thank you Jarek for the detailed input. I've taken some time to digest your
>> points before responding.
>>
>>
>>
>>
>> You've outlined a bold vision for Airflow 3, and I agree that being
>> decisive about the features and architectures will be the key to success.
>> However, before we make final decisions on what features to cut or retain,
>> it would be beneficial to have a more comprehensive understanding of how
>> the current features are utilized by the open-source community.
>>
>>
>>
>>
>> @Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io> <mailto:
>> kaxil@astronomer.io <ma...@astronomer.io>>> recently initiated a
>> discussion on
>> collecting telemetry from open-source deployments:
>> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m <
>> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
>> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
>> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;>.
>> This data
>> could be critical in ensuring our decisions are well-informed and reflect
>> the real-world usage patterns of our users, not just those from managed
>> environments like Astro, MWAA or GCC.
>>
>>
>>
>>
>> It's essential that we challenge our assumptions and base our decisions on
>> a holistic view of feature usage. Identifying potential cuts is a critical
>> step, but let's ensure our strategy aligns with the needs and preferences
>> of the broader Airflow community.
>>
>>
>>
>>
>> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <jarek@potiuk.com <mailto:
>> jarek@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
>> wrote:
>>
>>
>>
>>
>> > I am currently on sick leave, and still recovering - hoping to be able to
>> > travel next week to the US as planned, so I just wanted to break out of
>> it
>> > to make one comment here.
>> >
>> > I got a clearer head now a bit with medications hopefully working. I am
>> > still taking it that should help me to get over the current state, and I
>> > wanted to take a look at this discussion unraveling first. Over last week
>> > I disconnected from "day-to-day" Airflow and put some thoughts (as much
>> as
>> > I could in my current state) on it. The whole subject of this thread was
>> > started from that - how the current discussions on AIP-67 and others
>> change
>> > if we consider Airflow 3 is "starting".
>> >
>> > The price for back-compat is speed of development and quality. More
>> > combinations to test, more unexpected issues uncovered, necessity to keep
>> > parallel paths (old/new) while adding new features. All what Constance
>> > wrote about and what Ash explained. We already started to trip over our
>> own
>> > feet mutliple times in a few last releases. Have we tested all
>> combinations
>> > of deployment in Airflow 2.8 and 2.9 - not really, I think we already see
>> > that in a number of "combos" of features things are not working in as
>> > stable a way as they did before.
>> >
>> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a long
>> > time (or even move out) as they will not want to move to Airflow 3. A lot
>> > of the work implemented in AIP-44 and design of AIP-67 was done around
>> > back-compatibility. but yes -
>> > it would have been way easier if designed anew without back-compatibility
>> > in mind. And if we implement it and release it in Airflow 2 it will make
>> > new Airflow feature development even harder. That's why I wanted to treat
>> > it as "tactical" solution - hoping that in Airflow 3 we can make it
>> > "properly" - and that's why I started the discussion here when I sensed
>> > that we are "close" to Airflow 3 discussion, because I wanted to see what
>> > options we have there. This is why I have not yet concluded voting on
>> > AIP-67 waiting for the result of this discussion here.
>> >
>> > But if we are ready to go for Airflow.3 then I'd say there are two
>> > important things that should be part of the vision.
>> >
>> > 1) *We should be far more opinionated and have far fewer options of
>> > running things in Airflow 3*. Even an order of magnitude more
>> opinionated.
>> > Make choices, stick to it, perfect those opinionated choices to suit
>> 80/20
>> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not fitting
>> > the 20% that might choose to stay at Airflow 2. We can choose now which
>> > ~20% of cases we do not want to handle deliberately. And we should be
>> very,
>> > very strict about it. Default should be "no choice". This will radically
>> > simplify deployment and should make it easier to simplify Airflow
>> > development and DAG authoring experience because we will have less cases
>> to
>> > support. Even if we plan to add more options in the future, the first
>> > version of Airflow 3 should support one deployment approach only. This is
>> > the only way we can deliver it fast. And we should be very bold there.
>> > Choose one option and go for it in pretty much every place we have
>> choices
>> > now. We should Aim for Airflow 3.0 to support only a subset of current
>> > users - but those who are most likely to migrate first and those with the
>> > biggest need for the new features. We can think 3.x to support more
>> cases,
>> > but 3.0 should be as opinionated as humanly possible.
>> >
>> > And this deployment option should be also something ALL our stakeholders
>> > will feel OK with as a way forward in their offering.
>> >
>> > My candidates (and yes, some are bold):
>> >
>> > * *Drop MySQL*. If we have a single thing that makes us avoid our schema
>> > and DB migration - this is the case. Let's choose Postgres 15+ and use
>> some
>> > of the great features there. This will also enable much faster async SQL
>> > implementation and a number of other optimisations - not to mention
>> cutting
>> > every single change in development and testing time by literally half.
>> And
>> > we should not look back to adding MySQL.
>> > * *Drop Celery/Sequential Executor* and start with Local + K8S only (and
>> > AWS/Google others can continue developing theirs of course in parallel
>> and
>> > continue Hybrid executor work). Later - we figure out a better solution
>> to
>> > support "small" tasks using some new K8S features and possibly non-k8s
>> > solutions (Ray-based?)
>> > * *Cut Connection and Variable Management from DB/UI*. Leave only Secrets
>> > Management. Later when we have a 100% extensible React UI, we can add a
>> > "local DB secrets manager" add-on
>> > * *Choose a single way for DAG storage that will support versioning from
>> > day one*. Bear in mind we can add others later. Bolke's idea of using
>> > FSspec is an interesting one, we should see if it is feasible.
>> > * *Drop FAB completely (including custom plugins) and invest in
>> > implementing Auth Manager based on a dedicated, external solution*
>> > (KeyCloak
>> > that we've discussed before as a likely candidate)
>> > * *Leave Providers with Airflow 2 and add tests to make sure they are
>> > Airflow 3 future-compatible *- develop a way where we continue
>> development
>> > and contributions for Providers with Airflow 2 and add complete tests to
>> > run them with Airflow 3. This way we can continue developing Provider
>> > features independently, and make them work for Airflow 2 (and continue
>> > adding features for Airflow 2 users alongside Airflow 2 bugfixes), while
>> > also gradually fix any Airflow3 incompatibilities and instead of
>> > "back-compatibility" tests make provider "forward-compatibility" tests so
>> > that future Providers are tested and work on Airflow 3. Also it will make
>> > it easiest to continue Airflow 2 (bugfixes) + Providers tested without
>> > investing in changing the current CI / test harness.
>> > * *Simplify Test Harness for Airflow 3 from the start *- without
>> providers
>> > and 790+ dependencies, we could vastly simplify Airflow3 testing
>> (basically
>> > make CI jobs from scratch) using mostly standard Python tooling (while we
>> > can continue making use of the current test harness for Airflow 2 +
>> > Providers and extend it with Airflow 3 future-compatibility tests). That
>> > means Breeze would be only staying in Airflow 2 + Providers repo as we
>> > should be able to achieve most of what we have there with local venv/
>> > tooling (especially with uv as underlying tooling).
>> >
>> > 2) *I think we only add very few new "important" features. *Absolute
>> > minimum to make Airflow 3 appealing and add them only in Airflow 3:
>> > versioning, multi-team, pluggable UI should only be Airflow 3 - it makes
>> no
>> > sense to invest into Airflow 2 if we already know Airflow 3 is coming -
>> > that generally triples effort needed to get them out. We should drop new
>> > features development in Airflow 2. This will give users incentive to move
>> > to 3 if the new features will be worth it. Even paying
>> > compatibility/migration price.
>> >
>> > Versionig, for example: I believe if we decide to go only with Airflow 3
>> > and cut some of the above (Postgres only, Single versioning DAG storage)
>> we
>> > can make bolder decisions in versioning and support simpler models from
>> the
>> > get go (and deliver it faster). And we should add only a few - but
>> > important - features that our users clearly asked for and focus on
>> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or
>> 2.11).
>> > Similarly - multi-team can be simplified if we cut things from the list
>> > above and have Task isolation as first-class citizens in Airflow (and the
>> > only option).
>> >
>> > My candidates very much concur with the list shared by Kaxil in the doc +
>> > I'd add multi-team (but simplified thanks to the cuts). But I also here
>> > would mostly revert to Astronomer, Google. AWS team to define
>> collectively
>> > what is the absolute minimum set of features that would get the "target"
>> > part of their customers happy. And ONLY do that.
>> >
>> > So in short - I think the big part of our discussion should be what we
>> are
>> > ready to drop when we start airflow 3 and be very bold. Once we know we
>> > should figure out the absolute minimum of things that we can add that
>> will
>> > benefit a significant part of our users (and make use of increased speed
>> > because we dropped things).
>> >
>> > J.
>> >
>> >
>> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau
>> > <constance@astronomer.io.inva <ma...@astronomer.io.inva>
>> <mailto:constance@astronomer.io.inva <ma...@astronomer.io.inva>>lid>
>> wrote:
>> >
>> > > Hi Michal,
>> > >
>> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate your
>> > > concerns about the migration overhead for our users with a major new
>> > > version and see the appeal in your suggestion to integrate many of the
>> > > proposed changes into Airflow 2 through separate AIPs. It’s a valid
>> point
>> > > and certainly aligns with the value of making incremental improvements.
>> > >
>> > > However, after looking closely at the enhancements outlined for Airflow
>> > 3,
>> > > I'm convinced they warrant a new major release. Here’s why:
>> > >
>> > > 1. *Core Architectural Changes:* We’re looking at foundational changes
>> > > with Airflow 3—like redefining task priorities, separating task
>> > > definition
>> > > and task execution, and new AIPs like DAG versioning. remote execution
>> > > and restricting database access from workers. These aren’t just
>> > > incremental
>> > > improvements but major shifts that will set the stage for the next
>> > > decade
>> > > of Airflow’s architecture. Grouping these changes into a major release
>> > > will
>> > > help us make these transitions more cleanly and with fewer constraints
>> > > from
>> > > past decisions.
>> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140
>> > deprecated
>> > > issues, and this will only grow if we continue without a major
>> > cleanup.
>> > > This makes it increasingly difficult to implement new features
>> > > effectively
>> > > while maintaining backward compatibility. A major release allows us to
>> > > address these issues head-on, reducing technical debt and paving the
>> > way
>> > > for a more robust platform.
>> > > 3. *Managing Breaking Changes:* Let’s take the example of restricting
>> > > database access from workers. It’s a necessary move for better
>> > security
>> > > and
>> > > also potentially scalability reasons (reduces DB load). Many users
>> > have
>> > > workflows that interact with the DB, either by using raw sql or by
>> > > leveraging a session object. We could implement this feature in
>> > Airflow
>> > > 2
>> > > and avoid breaking existing workflows by continuing to have the old
>> > > standard mode as default - much of the work is already done - but that
>> > > would mean supporting both the new secure mode and the old standard
>> > mode
>> > > indefinitely and design new features with the assumption that most
>> > will
>> > > continue using the old standard mode. With Airflow 3, we can make
>> > secure
>> > > mode the default or even the only option, simplifying implementation
>> > and
>> > > future development. This is just one example where it is feasible to
>> > > implement in Airflow 2, but is better if we release it under the
>> > > context of
>> > > Airflow 3.
>> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up
>> > > possibilities for handling workflows beyond batch processing. Features
>> > > like
>> > > real-time DAG execution through API and multi-language task support
>> > are
>> > > big
>> > > steps forward, significantly expanding Airflow’s utility.
>> > >
>> > >
>> > > While integrating these updates into Airflow 2 might look less
>> disruptive
>> > > initially, the scale and nature of the required changes really support
>> a
>> > > move to Airflow 3. It’s not just about adding new features; it’s about
>> > > setting up Airflow so that it continues to remain relevant for the next
>> > ten
>> > > years.
>> > >
>> > > Constance
>> > >
>> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <ash@apache.org
>> <ma...@apache.org> <mailto:ash@apache.org <ma...@apache.org>>>
>> wrote:
>> > >
>> > > > There's a lot of technical debt hiding in Airflow, especially the
>> > > > scheduler that makes it harder and harder to efficiently add new
>> > > features.
>> > > >
>> > > > At some point, very soon, we are going to have to remove some very
>> > > > infrequently used back compat shims that negatively affect
>> performance.
>> > > > Without doing that the pace at which we can realistically add some of
>> > the
>> > > > more exciting features tends towards zero. Developer speed of
>> > > contributors
>> > > > is a factor here too!
>> > > >
>> > > > So while we are still using SemVer, that necessitates v3.
>> > > >
>> > > > Ash
>> > > >
>> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <michalmodras@google.com
>> <ma...@google.com> <mailto:michalmodras@google.com <mailto:
>> michalmodras@google.com>>
>> > > .INVALID>
>> > > > wrote:
>> > > > >+1 to Jens's & Bolke's points here and in the doc
>> > > > >
>> > > > >I agree we should work on clarifying the directions we would like
>> > > Airflow
>> > > > >to go. Introducing a new major Airflow version is a massive overhead
>> > for
>> > > > >users, who would need to plan for migrations, onboarding the new
>> > Airflow
>> > > > >(with a slightly different architecture), etc., and effectively
>> > Airflow
>> > > 2
>> > > > >would live in parallel for a long time.
>> > > > >
>> > > > >Personally, I think most of the points in Kaxil's/Vikram's doc are
>> > > > valuable
>> > > > >projects of their own, and I could imagine all of them being
>> delivered
>> > > as
>> > > > >separate AIPs within Airflow 2 (surely new minor versions of Airflow
>> > > 2). I
>> > > > >am not sure if the scope of changes and the goal we want to achieve
>> is
>> > > a)
>> > > > >clear enough b) broad enough to call for a new major version.
>> > > > >
>> > > > >Best,
>> > > > >Michal
>> > > > >
>> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T)
>> > > > ><Jens.Scheffler@de.bosch.com.inva <mailto:
>> Jens.Scheffler@de.bosch.com.inva> <mailto:Jens.Scheffler@de.bosch.com.inva
>> <ma...@de.bosch.com.inva>>lid> wrote:
>> > > > >
>> > > > >> Thanks for the document write-up, Kaxil. I assume this is mostly a
>> > > > vision
>> > > > >> statement.
>> > > > >>
>> > > > >> Looking forward for a larger addendum where we can collect things
>> > that
>> > > > we
>> > > > >> all can vote and agree on as targets.
>> > > > >>
>> > > > >> As I started earlier with a confluence page and it seems this is
>> not
>> > > > >> accessible to all, shall we convert this to a Google Doc for
>> better
>> > > > >> collaboration and item collection?
>> > > > >>
>> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> <
>> https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&gt;> <
>> https://aka.ms/o0ukef&amp;gt;&gt;>
>> > > > >> ________________________________
>> > > > >> From: Vikram Koka <vikram@astronomer.io.INVA <mailto:
>> vikram@astronomer.io.INVA> <mailto:vikram@astronomer.io.INVA <mailto:
>> vikram@astronomer.io.INVA>>LID>
>> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
>> > > > >> To: dev@airflow.apache.org <ma...@airflow.apache.org>
>> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>> <
>> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
>> dev@airflow.apache.org <ma...@airflow.apache.org>>>
>> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2)
>> vs
>> > > > >> strategic (Airflow 3) approach
>> > > > >>
>> > > > >> Thank you for your feedback, Bolke and Andrey!
>> > > > >>
>> > > > >> Bolke,
>> > > > >> I have replied to some of your comments in the doc.
>> > > > >> I will provide a detailed write up on the "Interactive DAG run"
>> (or
>> > > > >> synchronous DAG run) capability, which has generated some early
>> > > > questions.
>> > > > >> I had intended to get an AIP published for that as a follow-up,
>> but
>> > I
>> > > > >> believe that a simpler write up would be useful ahead of the AIP.
>> > > > >>
>> > > > >> Andrey,
>> > > > >> You raise an interesting point.
>> > > > >>
>> > > > >> As part of the Airflow 2.0 release, we as a community had decided
>> to
>> > > > >> strictly adhere to Semver as detailed in the document you
>> > referenced.
>> > > We
>> > > > >> also consciously split out the "Core Airflow" releases from the
>> > > > "Provider"
>> > > > >> releases at that time. We had a clear expectation then for the
>> > cadence
>> > > > of
>> > > > >> both minor and patch releases, which we have generally adhered to
>> > > since
>> > > > >> then.
>> > > > >>
>> > > > >> Personally, I am more concerned about our Provider releases right
>> > now,
>> > > > as
>> > > > >> compared to the cadence of our major releases. I believe that one
>> of
>> > > the
>> > > > >> proposed changes in the Airflow 3 document i.e. the clear
>> separation
>> > > for
>> > > > >> Task Execution will help here, but more may be needed.
>> > > > >>
>> > > > >> Definitely interested in more feedback on this as well.
>> > > > >>
>> > > > >> Vikram
>> > > > >>
>> > > > >>
>> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
>> > > andrey.anshin@taragol.is <ma...@taragol.is> <mailto:
>> andrey.anshin@taragol.is <ma...@taragol.is>>
>> > > > >
>> > > > >> wrote:
>> > > > >>
>> > > > >> > I would like to propose to change (at least discuss) release
>> > policy
>> > > > >> around
>> > > > >> > the Major version of Airflow.
>> > > > >> >
>> > > > >> > Right now it is described as "These releases do not happen with
>> > any
>> > > > >> regular
>> > > > >> > interval or on any predictable schedule." :
>> > > > >> >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
>> <
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0>
>> <
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0>
>> <
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt
>> ;>
>> > > > >> <
>> > > > >>
>> > > >
>> > >
>> >
>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
>> <
>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release>
>> <
>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release>
>> <
>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt
>> ;>
>> > > > >> >
>> > > > >> >
>> > > > >> > So maybe it is time to make it schedulable, e.g. one per two
>> years
>> > > or
>> > > > so.
>> > > > >> > This one could help us to avoid such a discussion in the future,
>> > > like
>> > > > "We
>> > > > >> > don't know when Airflow 4 is coming.". At the moment when the
>> new
>> > > > major
>> > > > >> > version will be released new features wouldn't be added in the
>> old
>> > > > major
>> > > > >> > version, however we would support bug / security for a while,
>> > e.g. 1
>> > > > year
>> > > > >> > for bug fixes, 3 years for security fixes with a total 5 year
>> > > > lifecycle
>> > > > >> per
>> > > > >> > a major version. These just are approximate time periods for a
>> > > > definition
>> > > > >> > of current period, bugfix period and security fix period.
>> > > > >> >
>> > > > >> > In contributors' perspective it helps with dropping the
>> deprecated
>> > > > stuff
>> > > > >> > which resolves some old problem: we have to support everything
>> > > > including
>> > > > >> > deprecated stuff and without schedulable lifecycle for the
>> > > deprecated
>> > > > >> stuff
>> > > > >> > it could be showstopper for the new feature, because sometimes
>> it
>> > > > hard to
>> > > > >> > support two different approaches for long period of time with no
>> > > hope
>> > > > >> that
>> > > > >> > it will happen soon. For some fundamental stuff which do not
>> > > require a
>> > > > >> lot
>> > > > >> > things time to support we could postponed removal for next after
>> > the
>> > > > next
>> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
>> > > > >> >
>> > > > >> > In the user perspective, they have at least bug fix support for
>> a
>> > > > while,
>> > > > >> if
>> > > > >> > someone want to use legacy version it their choice, however no
>> new
>> > > > >> > features, no new version of providers (after one year)
>> > > > >> >
>> > > > >> >
>> > > > >> > ----
>> > > > >> > Best Wishes
>> > > > >> > *Andrey Anshin*
>> > > > >> >
>> > > > >> >
>> > > > >> >
>> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bdbruin@gmail.com
>> <ma...@gmail.com> <mailto:bdbruin@gmail.com <mailto:
>> bdbruin@gmail.com>>>
>> > > > wrote:
>> > > > >> >
>> > > > >> > > I have left several comments :-). And on interactive dag runs
>> > even
>> > > > >> after
>> > > > >> > > the explanation of Vikram I still don't have a clue what we
>> want
>> > > to
>> > > > >> > > accomplish there :-P.
>> > > > >> > >
>> > > > >> > > I would like to see a mantra or team for Airflow 3. That helps
>> > > > nudging
>> > > > >> > > people in the same direction. Suggestions in the comments.
>> > > > >> > >
>> > > > >> > > Bolke
>> > > > >> > > Sent from my iPhone
>> > > > >> > >
>> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
>> > > <vikram@astronomer.io.inva <ma...@astronomer.io.inva> <mailto:
>> vikram@astronomer.io.inva <ma...@astronomer.io.inva>>lid
>> > > > >
>> > > > >> > > wrote:
>> > > > >> > > >
>> > > > >> > > > Good point Jed.
>> > > > >> > > > I responded back to your comment in the doc as well and very
>> > > open
>> > > > to
>> > > > >> > > > changing the term in the doc.
>> > > > >> > > >
>> > > > >> > > > Used the term "interactive DAG run" as the ability to invoke
>> > or
>> > > > >> > trigger a
>> > > > >> > > > DAG run through the API, with the expectation of getting
>> back
>> > a
>> > > > >> result
>> > > > >> > > > immediately. An alternate term could be a "synchronous DAG
>> > run".
>> > > > >> > > >
>> > > > >> > > > Regardless, this is a significant change so a good term to
>> > > > indicate
>> > > > >> the
>> > > > >> > > > expansion from "batch runs only" is warranted. Very open to
>> > > > different
>> > > > >> > > terms
>> > > > >> > > > here.
>> > > > >> > > >
>> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
>> > > > >> > jedcunningham@apache.org <ma...@apache.org>
>> <mailto:jedcunningham@apache.org <ma...@apache.org>>
>> > > > >> > > >
>> > > > >> > > >> wrote:
>> > > > >> > > >>
>> > > > >> > > >> Very exciting! Looks like we will have a busy period of
>> time
>> > > > ahead
>> > > > >> of
>> > > > >> > > us.
>> > > > >> > > >> Overall I like the plan so far, especially using this
>> year's
>> > > > Airflow
>> > > > >> > > Summit
>> > > > >> > > >> as an opportunity to announce and gather feedback, and the
>> > 2025
>> > > > >> > version
>> > > > >> > > to
>> > > > >> > > >> pitch upgrading.
>> > > > >> > > >>
>> > > > >> > > >> I left a comment in the doc, but we might want to iterate
>> on
>> > > the
>> > > > >> > > >> terminology we use for high priority or "synchronous" DAG
>> > runs
>> > > to
>> > > > >> > serve
>> > > > >> > > LLM
>> > > > >> > > >> responses - I find "interactive DAG runs" a bit confusing.
>> > > > >> > > >>
>> > > > >> > >
>> > > > >> > >
>> > > > ---------------------------------------------------------------------
>> > > > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> <ma...@airflow.apache.org> <mailto:
>> dev-unsubscribe@airflow.apache.org <mailto:
>> dev-unsubscribe@airflow.apache.org>>
>> > > > >> > > For additional commands, e-mail: dev-help@airflow.apache.org
>> <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org
>> <ma...@airflow.apache.org>>
>> > > > >> > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>>
>>
>>
>>
>>
>>
>>
>>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Jarek Potiuk <ja...@potiuk.com>]

I hope more voices will be coming soon as well :).

Constance - few words of explanation, for my "remove" things. I do not say
we "have to" remove those things or even that we "should".

My main motivation for mentioning the list is that we often forget that
when we have a new major version of software - removing stuff is as
important as adding. So I think when it comes to final decisions about
Airflow 3 we should be well aware about both: what we add and what we
remove. So my main point is - "removal" of things should be as important a
part in our future discussions as "adding", and I'd say with 10 years of
Airflow history, it's more important to remove things than to add them.

Side comment: Ash for one mentioned at multiple occasions how proud he is
that his contributions to Airflow are net-negative - he removed more code
than added. Unfortunately it's not shown any more here
https://github.com/apache/airflow/graphs/contributors - because we have
more than 10.000 commits, but I was hoping one day when we remove providers
from main development, I can do it personally and beat Ash to it.

Just wanted to stress a few things on why I think "discussing potential
removals" is important:

* I think it is super important to deliver Airflow 3 really fast. It took >
1.5 year to release Airflow 2 and it was far too long and keeping Airflow 1
and 2 in parallel was a huge pain and drain and it slowed us down
enormously. We should absolutely limit the time when we actively develop
both Airflow 2 features and Airflow 3 features.
* I think most of the current deprecations of the 100+ we have does not
slow us down AT ALL. Removal of those is mostly cosmetic change, a little
bit clutter to remove but they have little-to-no impact on actual speed of
development, it's just an old code that keeps on being around
* on the other hand - some of the things I mentioned ARE slowing us down A
LOT and will continue to do so until we remove them. For example, I believe
"postgresql + mysql + sqlite complete versioning solution for all the
possible variants of storage" will be quite a bit more complex and testing
will take far more time than if we drop mysql and choose a single storage
approach for Airflow 3.0. I have no hard data to back it up, but my gut
feeling is that it can take at least twice as long to get it out in the
hands of our users if we try to have Airflow 2 parity in Airflow 3.0 for
all the options we have there.
* the telemetry will be cool - but even if we add it for 2.10, the first
time meaningful data will be available is maybe 2 years from now when we do
Airflow 4. For now we can completely forget about it because it's only
going to show us some early adopters of Airflow 2.10. But we have surveys +
we can collect data from Astronomer, Google. Amazon for some usage and
identify who will be early adopters of Airflow 3 and target them first.
* the fact that we will drop something for Airflow 3.0, does not mean that
we have to drop it "forever". We can safely assume that Airflow 3.0 will be
only used by early adopters, and many of our users will wait for 3.1, 3.2
or ... 3.5 or 3.10 to migrate. We do not HAVE TO support all those "slower
moving users" from the 3.0. We can re-add things in 3.1 building on
foundations of 3.0 after it is in the hands of those early adopters. I
personally think we should prioritize speed of delivery of 3.0 over
"complete support of every deployment option of Airflow 2" - to allow
**some** of our early adopter users to migrate quickly, and add what's
missing for those who would like to move later in later versions.
* I think - this is the most important "product management" decision here -
based on the data we have available. Which users do we target for Airflow
3.0, and what we can drop to deliver it faster (and possibly re-add more
stuff for 3.1+). I think it's crucial to the success of the Airflow 3.0
initiative and the most important decision from the Product management side
that will impact the timeline of Airflow 3.0. While a lot depends on how
much time and effort individuals will spend on implementing things, the
decision of what we drop will impact everyone's speed and overall delivery
of Airflow 3.0. It's a decision we should not take lightly.

So my main point is: let's add "what we remove" as a very important point
in the product discussions we are going to have.

J.




On Wed, May 8, 2024 at 4:22 PM Bishundeo, Rajeshwar
<rb...@amazon.com.invalid> wrote:

> Like Constance and many others, there was time needed to process this
> fantastical summary and approach to Airflow 3. __
> A lot of the points raised by Jarek make sense, ie being prescriptive on
> what features we want to be there on Day 1 vs launched on 3.x - this does
> allow us to move quickly for our users.
> In line with Constance's concern, we need to be mindful of what we decide
> to keep and what we are willing to cut - which could be a painstaking
> process that could offset any gains of trying to be faster. I also believe
> that bringing all these new amazing features on Airflow 3 will peak the
> interest of early adopters and eventually get others interested in
> migration. However, I believe this migration will be a slow process and
> will present a gap in certain functionalities that users may want before
> entertaining any move to Airflow 3. There are still a lot of folks using
> v1.10 today. There were several tactical initiatives in the past few months
> with intent on bringing new functionality, ie Multi-team, to Airflow 2.x
> and I feel that while these efforts are not wasted, there should still be
> an option to continue improving Airflow 2 to avoid alienating our users on
> the basis of a future promise in Airflow 3, that may not be easy to migrate
> towards.
>
> -- Rajesh
>
>
>
>
>
>
>
>
> dev@airflow.apache.org <ma...@airflow.apache.org> <
> dev@airflow.apache.org <ma...@airflow.apache.org>>
>
>
> On 2024-05-07, 12:26 PM, "Constance Martineau"
> <constance@astronomer.io.INVA <ma...@astronomer.io.INVA>
> <mailto:constance@astronomer.io.INVA <ma...@astronomer.io.INVA>>LID>
> wrote:
>
>
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
>
>
>
>
>
>
> Thank you Jarek for the detailed input. I've taken some time to digest your
> points before responding.
>
>
>
>
> You've outlined a bold vision for Airflow 3, and I agree that being
> decisive about the features and architectures will be the key to success.
> However, before we make final decisions on what features to cut or retain,
> it would be beneficial to have a more comprehensive understanding of how
> the current features are utilized by the open-source community.
>
>
>
>
> @Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io> <mailto:
> kaxil@astronomer.io <ma...@astronomer.io>>> recently initiated a
> discussion on
> collecting telemetry from open-source deployments:
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m <
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <
> https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;>.
> This data
> could be critical in ensuring our decisions are well-informed and reflect
> the real-world usage patterns of our users, not just those from managed
> environments like Astro, MWAA or GCC.
>
>
>
>
> It's essential that we challenge our assumptions and base our decisions on
> a holistic view of feature usage. Identifying potential cuts is a critical
> step, but let's ensure our strategy aligns with the needs and preferences
> of the broader Airflow community.
>
>
>
>
> On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <jarek@potiuk.com <mailto:
> jarek@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>>
> wrote:
>
>
>
>
> > I am currently on sick leave, and still recovering - hoping to be able to
> > travel next week to the US as planned, so I just wanted to break out of
> it
> > to make one comment here.
> >
> > I got a clearer head now a bit with medications hopefully working. I am
> > still taking it that should help me to get over the current state, and I
> > wanted to take a look at this discussion unraveling first. Over last week
> > I disconnected from "day-to-day" Airflow and put some thoughts (as much
> as
> > I could in my current state) on it. The whole subject of this thread was
> > started from that - how the current discussions on AIP-67 and others
> change
> > if we consider Airflow 3 is "starting".
> >
> > The price for back-compat is speed of development and quality. More
> > combinations to test, more unexpected issues uncovered, necessity to keep
> > parallel paths (old/new) while adding new features. All what Constance
> > wrote about and what Ash explained. We already started to trip over our
> own
> > feet mutliple times in a few last releases. Have we tested all
> combinations
> > of deployment in Airflow 2.8 and 2.9 - not really, I think we already see
> > that in a number of "combos" of features things are not working in as
> > stable a way as they did before.
> >
> > Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a long
> > time (or even move out) as they will not want to move to Airflow 3. A lot
> > of the work implemented in AIP-44 and design of AIP-67 was done around
> > back-compatibility. but yes -
> > it would have been way easier if designed anew without back-compatibility
> > in mind. And if we implement it and release it in Airflow 2 it will make
> > new Airflow feature development even harder. That's why I wanted to treat
> > it as "tactical" solution - hoping that in Airflow 3 we can make it
> > "properly" - and that's why I started the discussion here when I sensed
> > that we are "close" to Airflow 3 discussion, because I wanted to see what
> > options we have there. This is why I have not yet concluded voting on
> > AIP-67 waiting for the result of this discussion here.
> >
> > But if we are ready to go for Airflow.3 then I'd say there are two
> > important things that should be part of the vision.
> >
> > 1) *We should be far more opinionated and have far fewer options of
> > running things in Airflow 3*. Even an order of magnitude more
> opinionated.
> > Make choices, stick to it, perfect those opinionated choices to suit
> 80/20
> > (or even 70/30 or maybe even 60/40) rule if you will. Risking not fitting
> > the 20% that might choose to stay at Airflow 2. We can choose now which
> > ~20% of cases we do not want to handle deliberately. And we should be
> very,
> > very strict about it. Default should be "no choice". This will radically
> > simplify deployment and should make it easier to simplify Airflow
> > development and DAG authoring experience because we will have less cases
> to
> > support. Even if we plan to add more options in the future, the first
> > version of Airflow 3 should support one deployment approach only. This is
> > the only way we can deliver it fast. And we should be very bold there.
> > Choose one option and go for it in pretty much every place we have
> choices
> > now. We should Aim for Airflow 3.0 to support only a subset of current
> > users - but those who are most likely to migrate first and those with the
> > biggest need for the new features. We can think 3.x to support more
> cases,
> > but 3.0 should be as opinionated as humanly possible.
> >
> > And this deployment option should be also something ALL our stakeholders
> > will feel OK with as a way forward in their offering.
> >
> > My candidates (and yes, some are bold):
> >
> > * *Drop MySQL*. If we have a single thing that makes us avoid our schema
> > and DB migration - this is the case. Let's choose Postgres 15+ and use
> some
> > of the great features there. This will also enable much faster async SQL
> > implementation and a number of other optimisations - not to mention
> cutting
> > every single change in development and testing time by literally half.
> And
> > we should not look back to adding MySQL.
> > * *Drop Celery/Sequential Executor* and start with Local + K8S only (and
> > AWS/Google others can continue developing theirs of course in parallel
> and
> > continue Hybrid executor work). Later - we figure out a better solution
> to
> > support "small" tasks using some new K8S features and possibly non-k8s
> > solutions (Ray-based?)
> > * *Cut Connection and Variable Management from DB/UI*. Leave only Secrets
> > Management. Later when we have a 100% extensible React UI, we can add a
> > "local DB secrets manager" add-on
> > * *Choose a single way for DAG storage that will support versioning from
> > day one*. Bear in mind we can add others later. Bolke's idea of using
> > FSspec is an interesting one, we should see if it is feasible.
> > * *Drop FAB completely (including custom plugins) and invest in
> > implementing Auth Manager based on a dedicated, external solution*
> > (KeyCloak
> > that we've discussed before as a likely candidate)
> > * *Leave Providers with Airflow 2 and add tests to make sure they are
> > Airflow 3 future-compatible *- develop a way where we continue
> development
> > and contributions for Providers with Airflow 2 and add complete tests to
> > run them with Airflow 3. This way we can continue developing Provider
> > features independently, and make them work for Airflow 2 (and continue
> > adding features for Airflow 2 users alongside Airflow 2 bugfixes), while
> > also gradually fix any Airflow3 incompatibilities and instead of
> > "back-compatibility" tests make provider "forward-compatibility" tests so
> > that future Providers are tested and work on Airflow 3. Also it will make
> > it easiest to continue Airflow 2 (bugfixes) + Providers tested without
> > investing in changing the current CI / test harness.
> > * *Simplify Test Harness for Airflow 3 from the start *- without
> providers
> > and 790+ dependencies, we could vastly simplify Airflow3 testing
> (basically
> > make CI jobs from scratch) using mostly standard Python tooling (while we
> > can continue making use of the current test harness for Airflow 2 +
> > Providers and extend it with Airflow 3 future-compatibility tests). That
> > means Breeze would be only staying in Airflow 2 + Providers repo as we
> > should be able to achieve most of what we have there with local venv/
> > tooling (especially with uv as underlying tooling).
> >
> > 2) *I think we only add very few new "important" features. *Absolute
> > minimum to make Airflow 3 appealing and add them only in Airflow 3:
> > versioning, multi-team, pluggable UI should only be Airflow 3 - it makes
> no
> > sense to invest into Airflow 2 if we already know Airflow 3 is coming -
> > that generally triples effort needed to get them out. We should drop new
> > features development in Airflow 2. This will give users incentive to move
> > to 3 if the new features will be worth it. Even paying
> > compatibility/migration price.
> >
> > Versionig, for example: I believe if we decide to go only with Airflow 3
> > and cut some of the above (Postgres only, Single versioning DAG storage)
> we
> > can make bolder decisions in versioning and support simpler models from
> the
> > get go (and deliver it faster). And we should add only a few - but
> > important - features that our users clearly asked for and focus on
> > delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or
> 2.11).
> > Similarly - multi-team can be simplified if we cut things from the list
> > above and have Task isolation as first-class citizens in Airflow (and the
> > only option).
> >
> > My candidates very much concur with the list shared by Kaxil in the doc +
> > I'd add multi-team (but simplified thanks to the cuts). But I also here
> > would mostly revert to Astronomer, Google. AWS team to define
> collectively
> > what is the absolute minimum set of features that would get the "target"
> > part of their customers happy. And ONLY do that.
> >
> > So in short - I think the big part of our discussion should be what we
> are
> > ready to drop when we start airflow 3 and be very bold. Once we know we
> > should figure out the absolute minimum of things that we can add that
> will
> > benefit a significant part of our users (and make use of increased speed
> > because we dropped things).
> >
> > J.
> >
> >
> > On Mon, May 6, 2024 at 8:40 PM Constance Martineau
> > <constance@astronomer.io.inva <ma...@astronomer.io.inva>
> <mailto:constance@astronomer.io.inva <ma...@astronomer.io.inva>>lid>
> wrote:
> >
> > > Hi Michal,
> > >
> > > Thanks for your thoughts on the Airflow 3 proposal. I appreciate your
> > > concerns about the migration overhead for our users with a major new
> > > version and see the appeal in your suggestion to integrate many of the
> > > proposed changes into Airflow 2 through separate AIPs. It’s a valid
> point
> > > and certainly aligns with the value of making incremental improvements.
> > >
> > > However, after looking closely at the enhancements outlined for Airflow
> > 3,
> > > I'm convinced they warrant a new major release. Here’s why:
> > >
> > > 1. *Core Architectural Changes:* We’re looking at foundational changes
> > > with Airflow 3—like redefining task priorities, separating task
> > > definition
> > > and task execution, and new AIPs like DAG versioning. remote execution
> > > and restricting database access from workers. These aren’t just
> > > incremental
> > > improvements but major shifts that will set the stage for the next
> > > decade
> > > of Airflow’s architecture. Grouping these changes into a major release
> > > will
> > > help us make these transitions more cleanly and with fewer constraints
> > > from
> > > past decisions.
> > > 2. *Code Clean-Up*: Our main branch has accumulated over 140
> > deprecated
> > > issues, and this will only grow if we continue without a major
> > cleanup.
> > > This makes it increasingly difficult to implement new features
> > > effectively
> > > while maintaining backward compatibility. A major release allows us to
> > > address these issues head-on, reducing technical debt and paving the
> > way
> > > for a more robust platform.
> > > 3. *Managing Breaking Changes:* Let’s take the example of restricting
> > > database access from workers. It’s a necessary move for better
> > security
> > > and
> > > also potentially scalability reasons (reduces DB load). Many users
> > have
> > > workflows that interact with the DB, either by using raw sql or by
> > > leveraging a session object. We could implement this feature in
> > Airflow
> > > 2
> > > and avoid breaking existing workflows by continuing to have the old
> > > standard mode as default - much of the work is already done - but that
> > > would mean supporting both the new secure mode and the old standard
> > mode
> > > indefinitely and design new features with the assumption that most
> > will
> > > continue using the old standard mode. With Airflow 3, we can make
> > secure
> > > mode the default or even the only option, simplifying implementation
> > and
> > > future development. This is just one example where it is feasible to
> > > implement in Airflow 2, but is better if we release it under the
> > > context of
> > > Airflow 3.
> > > 4. *Future-Proofing for New Features:* Airflow 3 will open up
> > > possibilities for handling workflows beyond batch processing. Features
> > > like
> > > real-time DAG execution through API and multi-language task support
> > are
> > > big
> > > steps forward, significantly expanding Airflow’s utility.
> > >
> > >
> > > While integrating these updates into Airflow 2 might look less
> disruptive
> > > initially, the scale and nature of the required changes really support
> a
> > > move to Airflow 3. It’s not just about adding new features; it’s about
> > > setting up Airflow so that it continues to remain relevant for the next
> > ten
> > > years.
> > >
> > > Constance
> > >
> > > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <ash@apache.org
> <ma...@apache.org> <mailto:ash@apache.org <ma...@apache.org>>>
> wrote:
> > >
> > > > There's a lot of technical debt hiding in Airflow, especially the
> > > > scheduler that makes it harder and harder to efficiently add new
> > > features.
> > > >
> > > > At some point, very soon, we are going to have to remove some very
> > > > infrequently used back compat shims that negatively affect
> performance.
> > > > Without doing that the pace at which we can realistically add some of
> > the
> > > > more exciting features tends towards zero. Developer speed of
> > > contributors
> > > > is a factor here too!
> > > >
> > > > So while we are still using SemVer, that necessitates v3.
> > > >
> > > > Ash
> > > >
> > > > On 6 May 2024 15:30:49 BST, "Michał Modras" <michalmodras@google.com
> <ma...@google.com> <mailto:michalmodras@google.com <mailto:
> michalmodras@google.com>>
> > > .INVALID>
> > > > wrote:
> > > > >+1 to Jens's & Bolke's points here and in the doc
> > > > >
> > > > >I agree we should work on clarifying the directions we would like
> > > Airflow
> > > > >to go. Introducing a new major Airflow version is a massive overhead
> > for
> > > > >users, who would need to plan for migrations, onboarding the new
> > Airflow
> > > > >(with a slightly different architecture), etc., and effectively
> > Airflow
> > > 2
> > > > >would live in parallel for a long time.
> > > > >
> > > > >Personally, I think most of the points in Kaxil's/Vikram's doc are
> > > > valuable
> > > > >projects of their own, and I could imagine all of them being
> delivered
> > > as
> > > > >separate AIPs within Airflow 2 (surely new minor versions of Airflow
> > > 2). I
> > > > >am not sure if the scope of changes and the goal we want to achieve
> is
> > > a)
> > > > >clear enough b) broad enough to call for a new major version.
> > > > >
> > > > >Best,
> > > > >Michal
> > > > >
> > > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> > > > ><Jens.Scheffler@de.bosch.com.inva <mailto:
> Jens.Scheffler@de.bosch.com.inva> <mailto:Jens.Scheffler@de.bosch.com.inva
> <ma...@de.bosch.com.inva>>lid> wrote:
> > > > >
> > > > >> Thanks for the document write-up, Kaxil. I assume this is mostly a
> > > > vision
> > > > >> statement.
> > > > >>
> > > > >> Looking forward for a larger addendum where we can collect things
> > that
> > > > we
> > > > >> all can vote and agree on as targets.
> > > > >>
> > > > >> As I started earlier with a confluence page and it seems this is
> not
> > > > >> accessible to all, shall we convert this to a Google Doc for
> better
> > > > >> collaboration and item collection?
> > > > >>
> > > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> <
> https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&gt;> <
> https://aka.ms/o0ukef&amp;gt;&gt;>
> > > > >> ________________________________
> > > > >> From: Vikram Koka <vikram@astronomer.io.INVA <mailto:
> vikram@astronomer.io.INVA> <mailto:vikram@astronomer.io.INVA <mailto:
> vikram@astronomer.io.INVA>>LID>
> > > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> > > > >> To: dev@airflow.apache.org <ma...@airflow.apache.org>
> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>> <
> dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:
> dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2)
> vs
> > > > >> strategic (Airflow 3) approach
> > > > >>
> > > > >> Thank you for your feedback, Bolke and Andrey!
> > > > >>
> > > > >> Bolke,
> > > > >> I have replied to some of your comments in the doc.
> > > > >> I will provide a detailed write up on the "Interactive DAG run"
> (or
> > > > >> synchronous DAG run) capability, which has generated some early
> > > > questions.
> > > > >> I had intended to get an AIP published for that as a follow-up,
> but
> > I
> > > > >> believe that a simpler write up would be useful ahead of the AIP.
> > > > >>
> > > > >> Andrey,
> > > > >> You raise an interesting point.
> > > > >>
> > > > >> As part of the Airflow 2.0 release, we as a community had decided
> to
> > > > >> strictly adhere to Semver as detailed in the document you
> > referenced.
> > > We
> > > > >> also consciously split out the "Core Airflow" releases from the
> > > > "Provider"
> > > > >> releases at that time. We had a clear expectation then for the
> > cadence
> > > > of
> > > > >> both minor and patch releases, which we have generally adhered to
> > > since
> > > > >> then.
> > > > >>
> > > > >> Personally, I am more concerned about our Provider releases right
> > now,
> > > > as
> > > > >> compared to the cadence of our major releases. I believe that one
> of
> > > the
> > > > >> proposed changes in the Airflow 3 document i.e. the clear
> separation
> > > for
> > > > >> Task Execution will help here, but more may be needed.
> > > > >>
> > > > >> Definitely interested in more feedback on this as well.
> > > > >>
> > > > >> Vikram
> > > > >>
> > > > >>
> > > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> > > andrey.anshin@taragol.is <ma...@taragol.is> <mailto:
> andrey.anshin@taragol.is <ma...@taragol.is>>
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >> > I would like to propose to change (at least discuss) release
> > policy
> > > > >> around
> > > > >> > the Major version of Airflow.
> > > > >> >
> > > > >> > Right now it is described as "These releases do not happen with
> > any
> > > > >> regular
> > > > >> > interval or on any predictable schedule." :
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0>
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0>
> <
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt
> ;>
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> <
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release>
> <
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release>
> <
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt
> ;>
> > > > >> >
> > > > >> >
> > > > >> > So maybe it is time to make it schedulable, e.g. one per two
> years
> > > or
> > > > so.
> > > > >> > This one could help us to avoid such a discussion in the future,
> > > like
> > > > "We
> > > > >> > don't know when Airflow 4 is coming.". At the moment when the
> new
> > > > major
> > > > >> > version will be released new features wouldn't be added in the
> old
> > > > major
> > > > >> > version, however we would support bug / security for a while,
> > e.g. 1
> > > > year
> > > > >> > for bug fixes, 3 years for security fixes with a total 5 year
> > > > lifecycle
> > > > >> per
> > > > >> > a major version. These just are approximate time periods for a
> > > > definition
> > > > >> > of current period, bugfix period and security fix period.
> > > > >> >
> > > > >> > In contributors' perspective it helps with dropping the
> deprecated
> > > > stuff
> > > > >> > which resolves some old problem: we have to support everything
> > > > including
> > > > >> > deprecated stuff and without schedulable lifecycle for the
> > > deprecated
> > > > >> stuff
> > > > >> > it could be showstopper for the new feature, because sometimes
> it
> > > > hard to
> > > > >> > support two different approaches for long period of time with no
> > > hope
> > > > >> that
> > > > >> > it will happen soon. For some fundamental stuff which do not
> > > require a
> > > > >> lot
> > > > >> > things time to support we could postponed removal for next after
> > the
> > > > next
> > > > >> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
> > > > >> >
> > > > >> > In the user perspective, they have at least bug fix support for
> a
> > > > while,
> > > > >> if
> > > > >> > someone want to use legacy version it their choice, however no
> new
> > > > >> > features, no new version of providers (after one year)
> > > > >> >
> > > > >> >
> > > > >> > ----
> > > > >> > Best Wishes
> > > > >> > *Andrey Anshin*
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bdbruin@gmail.com
> <ma...@gmail.com> <mailto:bdbruin@gmail.com <mailto:
> bdbruin@gmail.com>>>
> > > > wrote:
> > > > >> >
> > > > >> > > I have left several comments :-). And on interactive dag runs
> > even
> > > > >> after
> > > > >> > > the explanation of Vikram I still don't have a clue what we
> want
> > > to
> > > > >> > > accomplish there :-P.
> > > > >> > >
> > > > >> > > I would like to see a mantra or team for Airflow 3. That helps
> > > > nudging
> > > > >> > > people in the same direction. Suggestions in the comments.
> > > > >> > >
> > > > >> > > Bolke
> > > > >> > > Sent from my iPhone
> > > > >> > >
> > > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> > > <vikram@astronomer.io.inva <ma...@astronomer.io.inva> <mailto:
> vikram@astronomer.io.inva <ma...@astronomer.io.inva>>lid
> > > > >
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > Good point Jed.
> > > > >> > > > I responded back to your comment in the doc as well and very
> > > open
> > > > to
> > > > >> > > > changing the term in the doc.
> > > > >> > > >
> > > > >> > > > Used the term "interactive DAG run" as the ability to invoke
> > or
> > > > >> > trigger a
> > > > >> > > > DAG run through the API, with the expectation of getting
> back
> > a
> > > > >> result
> > > > >> > > > immediately. An alternate term could be a "synchronous DAG
> > run".
> > > > >> > > >
> > > > >> > > > Regardless, this is a significant change so a good term to
> > > > indicate
> > > > >> the
> > > > >> > > > expansion from "batch runs only" is warranted. Very open to
> > > > different
> > > > >> > > terms
> > > > >> > > > here.
> > > > >> > > >
> > > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > > > >> > jedcunningham@apache.org <ma...@apache.org>
> <mailto:jedcunningham@apache.org <ma...@apache.org>>
> > > > >> > > >
> > > > >> > > >> wrote:
> > > > >> > > >>
> > > > >> > > >> Very exciting! Looks like we will have a busy period of
> time
> > > > ahead
> > > > >> of
> > > > >> > > us.
> > > > >> > > >> Overall I like the plan so far, especially using this
> year's
> > > > Airflow
> > > > >> > > Summit
> > > > >> > > >> as an opportunity to announce and gather feedback, and the
> > 2025
> > > > >> > version
> > > > >> > > to
> > > > >> > > >> pitch upgrading.
> > > > >> > > >>
> > > > >> > > >> I left a comment in the doc, but we might want to iterate
> on
> > > the
> > > > >> > > >> terminology we use for high priority or "synchronous" DAG
> > runs
> > > to
> > > > >> > serve
> > > > >> > > LLM
> > > > >> > > >> responses - I find "interactive DAG runs" a bit confusing.
> > > > >> > > >>
> > > > >> > >
> > > > >> > >
> > > > ---------------------------------------------------------------------
> > > > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> <ma...@airflow.apache.org> <mailto:
> dev-unsubscribe@airflow.apache.org <mailto:
> dev-unsubscribe@airflow.apache.org>>
> > > > >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org
> <ma...@airflow.apache.org>>
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
>
>
>
>
>
>
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

["Bishundeo, Rajeshwar" <rb...@amazon.com.INVALID>]

Like Constance and many others, there was time needed to process this fantastical summary and approach to Airflow 3. __ 
A lot of the points raised by Jarek make sense, ie being prescriptive on what features we want to be there on Day 1 vs launched on 3.x - this does allow us to move quickly for our users.
In line with Constance's concern, we need to be mindful of what we decide to keep and what we are willing to cut - which could be a painstaking process that could offset any gains of trying to be faster. I also believe that bringing all these new amazing features on Airflow 3 will peak the interest of early adopters and eventually get others interested in migration. However, I believe this migration will be a slow process and will present a gap in certain functionalities that users may want before entertaining any move to Airflow 3. There are still a lot of folks using v1.10 today. There were several tactical initiatives in the past few months with intent on bringing new functionality, ie Multi-team, to Airflow 2.x and I feel that while these efforts are not wasted, there should still be an option to continue improving Airflow 2 to avoid alienating our users on the basis of a future promise in Airflow 3, that may not be easy to migrate towards.

-- Rajesh 








dev@airflow.apache.org <ma...@airflow.apache.org> <dev@airflow.apache.org <ma...@airflow.apache.org>>


On 2024-05-07, 12:26 PM, "Constance Martineau" <constance@astronomer.io.INVA <ma...@astronomer.io.INVA> <mailto:constance@astronomer.io.INVA <ma...@astronomer.io.INVA>>LID> wrote:




CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.












AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.












Thank you Jarek for the detailed input. I've taken some time to digest your
points before responding.




You've outlined a bold vision for Airflow 3, and I agree that being
decisive about the features and architectures will be the key to success.
However, before we make final decisions on what features to cut or retain,
it would be beneficial to have a more comprehensive understanding of how
the current features are utilized by the open-source community.




@Kaxil Naik <kaxil@astronomer.io <ma...@astronomer.io> <mailto:kaxil@astronomer.io <ma...@astronomer.io>>> recently initiated a discussion on
collecting telemetry from open-source deployments:
https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m> <https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m&gt;>. This data
could be critical in ensuring our decisions are well-informed and reflect
the real-world usage patterns of our users, not just those from managed
environments like Astro, MWAA or GCC.




It's essential that we challenge our assumptions and base our decisions on
a holistic view of feature usage. Identifying potential cuts is a critical
step, but let's ensure our strategy aligns with the needs and preferences
of the broader Airflow community.




On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com> <mailto:jarek@potiuk.com <ma...@potiuk.com>>> wrote:




> I am currently on sick leave, and still recovering - hoping to be able to
> travel next week to the US as planned, so I just wanted to break out of it
> to make one comment here.
>
> I got a clearer head now a bit with medications hopefully working. I am
> still taking it that should help me to get over the current state, and I
> wanted to take a look at this discussion unraveling first. Over last week
> I disconnected from "day-to-day" Airflow and put some thoughts (as much as
> I could in my current state) on it. The whole subject of this thread was
> started from that - how the current discussions on AIP-67 and others change
> if we consider Airflow 3 is "starting".
>
> The price for back-compat is speed of development and quality. More
> combinations to test, more unexpected issues uncovered, necessity to keep
> parallel paths (old/new) while adding new features. All what Constance
> wrote about and what Ash explained. We already started to trip over our own
> feet mutliple times in a few last releases. Have we tested all combinations
> of deployment in Airflow 2.8 and 2.9 - not really, I think we already see
> that in a number of "combos" of features things are not working in as
> stable a way as they did before.
>
> Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a long
> time (or even move out) as they will not want to move to Airflow 3. A lot
> of the work implemented in AIP-44 and design of AIP-67 was done around
> back-compatibility. but yes -
> it would have been way easier if designed anew without back-compatibility
> in mind. And if we implement it and release it in Airflow 2 it will make
> new Airflow feature development even harder. That's why I wanted to treat
> it as "tactical" solution - hoping that in Airflow 3 we can make it
> "properly" - and that's why I started the discussion here when I sensed
> that we are "close" to Airflow 3 discussion, because I wanted to see what
> options we have there. This is why I have not yet concluded voting on
> AIP-67 waiting for the result of this discussion here.
>
> But if we are ready to go for Airflow.3 then I'd say there are two
> important things that should be part of the vision.
>
> 1) *We should be far more opinionated and have far fewer options of
> running things in Airflow 3*. Even an order of magnitude more opinionated.
> Make choices, stick to it, perfect those opinionated choices to suit 80/20
> (or even 70/30 or maybe even 60/40) rule if you will. Risking not fitting
> the 20% that might choose to stay at Airflow 2. We can choose now which
> ~20% of cases we do not want to handle deliberately. And we should be very,
> very strict about it. Default should be "no choice". This will radically
> simplify deployment and should make it easier to simplify Airflow
> development and DAG authoring experience because we will have less cases to
> support. Even if we plan to add more options in the future, the first
> version of Airflow 3 should support one deployment approach only. This is
> the only way we can deliver it fast. And we should be very bold there.
> Choose one option and go for it in pretty much every place we have choices
> now. We should Aim for Airflow 3.0 to support only a subset of current
> users - but those who are most likely to migrate first and those with the
> biggest need for the new features. We can think 3.x to support more cases,
> but 3.0 should be as opinionated as humanly possible.
>
> And this deployment option should be also something ALL our stakeholders
> will feel OK with as a way forward in their offering.
>
> My candidates (and yes, some are bold):
>
> * *Drop MySQL*. If we have a single thing that makes us avoid our schema
> and DB migration - this is the case. Let's choose Postgres 15+ and use some
> of the great features there. This will also enable much faster async SQL
> implementation and a number of other optimisations - not to mention cutting
> every single change in development and testing time by literally half. And
> we should not look back to adding MySQL.
> * *Drop Celery/Sequential Executor* and start with Local + K8S only (and
> AWS/Google others can continue developing theirs of course in parallel and
> continue Hybrid executor work). Later - we figure out a better solution to
> support "small" tasks using some new K8S features and possibly non-k8s
> solutions (Ray-based?)
> * *Cut Connection and Variable Management from DB/UI*. Leave only Secrets
> Management. Later when we have a 100% extensible React UI, we can add a
> "local DB secrets manager" add-on
> * *Choose a single way for DAG storage that will support versioning from
> day one*. Bear in mind we can add others later. Bolke's idea of using
> FSspec is an interesting one, we should see if it is feasible.
> * *Drop FAB completely (including custom plugins) and invest in
> implementing Auth Manager based on a dedicated, external solution*
> (KeyCloak
> that we've discussed before as a likely candidate)
> * *Leave Providers with Airflow 2 and add tests to make sure they are
> Airflow 3 future-compatible *- develop a way where we continue development
> and contributions for Providers with Airflow 2 and add complete tests to
> run them with Airflow 3. This way we can continue developing Provider
> features independently, and make them work for Airflow 2 (and continue
> adding features for Airflow 2 users alongside Airflow 2 bugfixes), while
> also gradually fix any Airflow3 incompatibilities and instead of
> "back-compatibility" tests make provider "forward-compatibility" tests so
> that future Providers are tested and work on Airflow 3. Also it will make
> it easiest to continue Airflow 2 (bugfixes) + Providers tested without
> investing in changing the current CI / test harness.
> * *Simplify Test Harness for Airflow 3 from the start *- without providers
> and 790+ dependencies, we could vastly simplify Airflow3 testing (basically
> make CI jobs from scratch) using mostly standard Python tooling (while we
> can continue making use of the current test harness for Airflow 2 +
> Providers and extend it with Airflow 3 future-compatibility tests). That
> means Breeze would be only staying in Airflow 2 + Providers repo as we
> should be able to achieve most of what we have there with local venv/
> tooling (especially with uv as underlying tooling).
>
> 2) *I think we only add very few new "important" features. *Absolute
> minimum to make Airflow 3 appealing and add them only in Airflow 3:
> versioning, multi-team, pluggable UI should only be Airflow 3 - it makes no
> sense to invest into Airflow 2 if we already know Airflow 3 is coming -
> that generally triples effort needed to get them out. We should drop new
> features development in Airflow 2. This will give users incentive to move
> to 3 if the new features will be worth it. Even paying
> compatibility/migration price.
>
> Versionig, for example: I believe if we decide to go only with Airflow 3
> and cut some of the above (Postgres only, Single versioning DAG storage) we
> can make bolder decisions in versioning and support simpler models from the
> get go (and deliver it faster). And we should add only a few - but
> important - features that our users clearly asked for and focus on
> delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or 2.11).
> Similarly - multi-team can be simplified if we cut things from the list
> above and have Task isolation as first-class citizens in Airflow (and the
> only option).
>
> My candidates very much concur with the list shared by Kaxil in the doc +
> I'd add multi-team (but simplified thanks to the cuts). But I also here
> would mostly revert to Astronomer, Google. AWS team to define collectively
> what is the absolute minimum set of features that would get the "target"
> part of their customers happy. And ONLY do that.
>
> So in short - I think the big part of our discussion should be what we are
> ready to drop when we start airflow 3 and be very bold. Once we know we
> should figure out the absolute minimum of things that we can add that will
> benefit a significant part of our users (and make use of increased speed
> because we dropped things).
>
> J.
>
>
> On Mon, May 6, 2024 at 8:40 PM Constance Martineau
> <constance@astronomer.io.inva <ma...@astronomer.io.inva> <mailto:constance@astronomer.io.inva <ma...@astronomer.io.inva>>lid> wrote:
>
> > Hi Michal,
> >
> > Thanks for your thoughts on the Airflow 3 proposal. I appreciate your
> > concerns about the migration overhead for our users with a major new
> > version and see the appeal in your suggestion to integrate many of the
> > proposed changes into Airflow 2 through separate AIPs. It’s a valid point
> > and certainly aligns with the value of making incremental improvements.
> >
> > However, after looking closely at the enhancements outlined for Airflow
> 3,
> > I'm convinced they warrant a new major release. Here’s why:
> >
> > 1. *Core Architectural Changes:* We’re looking at foundational changes
> > with Airflow 3—like redefining task priorities, separating task
> > definition
> > and task execution, and new AIPs like DAG versioning. remote execution
> > and restricting database access from workers. These aren’t just
> > incremental
> > improvements but major shifts that will set the stage for the next
> > decade
> > of Airflow’s architecture. Grouping these changes into a major release
> > will
> > help us make these transitions more cleanly and with fewer constraints
> > from
> > past decisions.
> > 2. *Code Clean-Up*: Our main branch has accumulated over 140
> deprecated
> > issues, and this will only grow if we continue without a major
> cleanup.
> > This makes it increasingly difficult to implement new features
> > effectively
> > while maintaining backward compatibility. A major release allows us to
> > address these issues head-on, reducing technical debt and paving the
> way
> > for a more robust platform.
> > 3. *Managing Breaking Changes:* Let’s take the example of restricting
> > database access from workers. It’s a necessary move for better
> security
> > and
> > also potentially scalability reasons (reduces DB load). Many users
> have
> > workflows that interact with the DB, either by using raw sql or by
> > leveraging a session object. We could implement this feature in
> Airflow
> > 2
> > and avoid breaking existing workflows by continuing to have the old
> > standard mode as default - much of the work is already done - but that
> > would mean supporting both the new secure mode and the old standard
> mode
> > indefinitely and design new features with the assumption that most
> will
> > continue using the old standard mode. With Airflow 3, we can make
> secure
> > mode the default or even the only option, simplifying implementation
> and
> > future development. This is just one example where it is feasible to
> > implement in Airflow 2, but is better if we release it under the
> > context of
> > Airflow 3.
> > 4. *Future-Proofing for New Features:* Airflow 3 will open up
> > possibilities for handling workflows beyond batch processing. Features
> > like
> > real-time DAG execution through API and multi-language task support
> are
> > big
> > steps forward, significantly expanding Airflow’s utility.
> >
> >
> > While integrating these updates into Airflow 2 might look less disruptive
> > initially, the scale and nature of the required changes really support a
> > move to Airflow 3. It’s not just about adding new features; it’s about
> > setting up Airflow so that it continues to remain relevant for the next
> ten
> > years.
> >
> > Constance
> >
> > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org> <mailto:ash@apache.org <ma...@apache.org>>> wrote:
> >
> > > There's a lot of technical debt hiding in Airflow, especially the
> > > scheduler that makes it harder and harder to efficiently add new
> > features.
> > >
> > > At some point, very soon, we are going to have to remove some very
> > > infrequently used back compat shims that negatively affect performance.
> > > Without doing that the pace at which we can realistically add some of
> the
> > > more exciting features tends towards zero. Developer speed of
> > contributors
> > > is a factor here too!
> > >
> > > So while we are still using SemVer, that necessitates v3.
> > >
> > > Ash
> > >
> > > On 6 May 2024 15:30:49 BST, "Michał Modras" <michalmodras@google.com <ma...@google.com> <mailto:michalmodras@google.com <ma...@google.com>>
> > .INVALID>
> > > wrote:
> > > >+1 to Jens's & Bolke's points here and in the doc
> > > >
> > > >I agree we should work on clarifying the directions we would like
> > Airflow
> > > >to go. Introducing a new major Airflow version is a massive overhead
> for
> > > >users, who would need to plan for migrations, onboarding the new
> Airflow
> > > >(with a slightly different architecture), etc., and effectively
> Airflow
> > 2
> > > >would live in parallel for a long time.
> > > >
> > > >Personally, I think most of the points in Kaxil's/Vikram's doc are
> > > valuable
> > > >projects of their own, and I could imagine all of them being delivered
> > as
> > > >separate AIPs within Airflow 2 (surely new minor versions of Airflow
> > 2). I
> > > >am not sure if the scope of changes and the goal we want to achieve is
> > a)
> > > >clear enough b) broad enough to call for a new major version.
> > > >
> > > >Best,
> > > >Michal
> > > >
> > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> > > ><Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva> <mailto:Jens.Scheffler@de.bosch.com.inva <ma...@de.bosch.com.inva>>lid> wrote:
> > > >
> > > >> Thanks for the document write-up, Kaxil. I assume this is mostly a
> > > vision
> > > >> statement.
> > > >>
> > > >> Looking forward for a larger addendum where we can collect things
> that
> > > we
> > > >> all can vote and agree on as targets.
> > > >>
> > > >> As I started earlier with a confluence page and it seems this is not
> > > >> accessible to all, shall we convert this to a Google Doc for better
> > > >> collaboration and item collection?
> > > >>
> > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef> <https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&gt;> <https://aka.ms/o0ukef&amp;gt;&gt;>
> > > >> ________________________________
> > > >> From: Vikram Koka <vikram@astronomer.io.INVA <ma...@astronomer.io.INVA> <mailto:vikram@astronomer.io.INVA <ma...@astronomer.io.INVA>>LID>
> > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> > > >> To: dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>> <dev@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev@airflow.apache.org <ma...@airflow.apache.org>>>
> > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
> > > >> strategic (Airflow 3) approach
> > > >>
> > > >> Thank you for your feedback, Bolke and Andrey!
> > > >>
> > > >> Bolke,
> > > >> I have replied to some of your comments in the doc.
> > > >> I will provide a detailed write up on the "Interactive DAG run" (or
> > > >> synchronous DAG run) capability, which has generated some early
> > > questions.
> > > >> I had intended to get an AIP published for that as a follow-up, but
> I
> > > >> believe that a simpler write up would be useful ahead of the AIP.
> > > >>
> > > >> Andrey,
> > > >> You raise an interesting point.
> > > >>
> > > >> As part of the Airflow 2.0 release, we as a community had decided to
> > > >> strictly adhere to Semver as detailed in the document you
> referenced.
> > We
> > > >> also consciously split out the "Core Airflow" releases from the
> > > "Provider"
> > > >> releases at that time. We had a clear expectation then for the
> cadence
> > > of
> > > >> both minor and patch releases, which we have generally adhered to
> > since
> > > >> then.
> > > >>
> > > >> Personally, I am more concerned about our Provider releases right
> now,
> > > as
> > > >> compared to the cadence of our major releases. I believe that one of
> > the
> > > >> proposed changes in the Airflow 3 document i.e. the clear separation
> > for
> > > >> Task Execution will help here, but more may be needed.
> > > >>
> > > >> Definitely interested in more feedback on this as well.
> > > >>
> > > >> Vikram
> > > >>
> > > >>
> > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> > andrey.anshin@taragol.is <ma...@taragol.is> <mailto:andrey.anshin@taragol.is <ma...@taragol.is>>
> > > >
> > > >> wrote:
> > > >>
> > > >> > I would like to propose to change (at least discuss) release
> policy
> > > >> around
> > > >> > the Major version of Airflow.
> > > >> >
> > > >> > Right now it is described as "These releases do not happen with
> any
> > > >> regular
> > > >> > interval or on any predictable schedule." :
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0 <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;reserved=0> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&amp;amp;data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&amp;amp;sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&amp;amp;reserved=0&gt;>
> > > >> <
> > > >>
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release> <https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release&gt;>
> > > >> >
> > > >> >
> > > >> > So maybe it is time to make it schedulable, e.g. one per two years
> > or
> > > so.
> > > >> > This one could help us to avoid such a discussion in the future,
> > like
> > > "We
> > > >> > don't know when Airflow 4 is coming.". At the moment when the new
> > > major
> > > >> > version will be released new features wouldn't be added in the old
> > > major
> > > >> > version, however we would support bug / security for a while,
> e.g. 1
> > > year
> > > >> > for bug fixes, 3 years for security fixes with a total 5 year
> > > lifecycle
> > > >> per
> > > >> > a major version. These just are approximate time periods for a
> > > definition
> > > >> > of current period, bugfix period and security fix period.
> > > >> >
> > > >> > In contributors' perspective it helps with dropping the deprecated
> > > stuff
> > > >> > which resolves some old problem: we have to support everything
> > > including
> > > >> > deprecated stuff and without schedulable lifecycle for the
> > deprecated
> > > >> stuff
> > > >> > it could be showstopper for the new feature, because sometimes it
> > > hard to
> > > >> > support two different approaches for long period of time with no
> > hope
> > > >> that
> > > >> > it will happen soon. For some fundamental stuff which do not
> > require a
> > > >> lot
> > > >> > things time to support we could postponed removal for next after
> the
> > > next
> > > >> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
> > > >> >
> > > >> > In the user perspective, they have at least bug fix support for a
> > > while,
> > > >> if
> > > >> > someone want to use legacy version it their choice, however no new
> > > >> > features, no new version of providers (after one year)
> > > >> >
> > > >> >
> > > >> > ----
> > > >> > Best Wishes
> > > >> > *Andrey Anshin*
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com> <mailto:bdbruin@gmail.com <ma...@gmail.com>>>
> > > wrote:
> > > >> >
> > > >> > > I have left several comments :-). And on interactive dag runs
> even
> > > >> after
> > > >> > > the explanation of Vikram I still don't have a clue what we want
> > to
> > > >> > > accomplish there :-P.
> > > >> > >
> > > >> > > I would like to see a mantra or team for Airflow 3. That helps
> > > nudging
> > > >> > > people in the same direction. Suggestions in the comments.
> > > >> > >
> > > >> > > Bolke
> > > >> > > Sent from my iPhone
> > > >> > >
> > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> > <vikram@astronomer.io.inva <ma...@astronomer.io.inva> <mailto:vikram@astronomer.io.inva <ma...@astronomer.io.inva>>lid
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > Good point Jed.
> > > >> > > > I responded back to your comment in the doc as well and very
> > open
> > > to
> > > >> > > > changing the term in the doc.
> > > >> > > >
> > > >> > > > Used the term "interactive DAG run" as the ability to invoke
> or
> > > >> > trigger a
> > > >> > > > DAG run through the API, with the expectation of getting back
> a
> > > >> result
> > > >> > > > immediately. An alternate term could be a "synchronous DAG
> run".
> > > >> > > >
> > > >> > > > Regardless, this is a significant change so a good term to
> > > indicate
> > > >> the
> > > >> > > > expansion from "batch runs only" is warranted. Very open to
> > > different
> > > >> > > terms
> > > >> > > > here.
> > > >> > > >
> > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > > >> > jedcunningham@apache.org <ma...@apache.org> <mailto:jedcunningham@apache.org <ma...@apache.org>>
> > > >> > > >
> > > >> > > >> wrote:
> > > >> > > >>
> > > >> > > >> Very exciting! Looks like we will have a busy period of time
> > > ahead
> > > >> of
> > > >> > > us.
> > > >> > > >> Overall I like the plan so far, especially using this year's
> > > Airflow
> > > >> > > Summit
> > > >> > > >> as an opportunity to announce and gather feedback, and the
> 2025
> > > >> > version
> > > >> > > to
> > > >> > > >> pitch upgrading.
> > > >> > > >>
> > > >> > > >> I left a comment in the doc, but we might want to iterate on
> > the
> > > >> > > >> terminology we use for high priority or "synchronous" DAG
> runs
> > to
> > > >> > serve
> > > >> > > LLM
> > > >> > > >> responses - I find "interactive DAG runs" a bit confusing.
> > > >> > > >>
> > > >> > >
> > > >> > >
> > > ---------------------------------------------------------------------
> > > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-unsubscribe@airflow.apache.org <ma...@airflow.apache.org>>
> > > >> > > For additional commands, e-mail: dev-help@airflow.apache.org <ma...@airflow.apache.org> <mailto:dev-help@airflow.apache.org <ma...@airflow.apache.org>>
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Constance Martineau <co...@astronomer.io.INVALID>]

Thank you Jarek for the detailed input. I've taken some time to digest your
points before responding.

You've outlined a bold vision for Airflow 3, and I agree that being
decisive about the features and architectures will be the key to success.
However, before we make final decisions on what features to cut or retain,
it would be beneficial to have a more comprehensive understanding of how
the current features are utilized by the open-source community.

@Kaxil Naik <ka...@astronomer.io> recently initiated a discussion on
collecting telemetry from open-source deployments:
https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m. This data
could be critical in ensuring our decisions are well-informed and reflect
the real-world usage patterns of our users, not just those from managed
environments like Astro, MWAA or GCC.

It's essential that we challenge our assumptions and base our decisions on
a holistic view of feature usage. Identifying potential cuts is a critical
step, but let's ensure our strategy aligns with the needs and preferences
of the broader Airflow community.

On Mon, May 6, 2024 at 6:50 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> I am currently on sick leave, and still recovering - hoping to be able to
> travel next week to the US as planned, so I just wanted to break out of it
> to make one comment here.
>
> I got a clearer head now a bit with medications hopefully working. I am
> still taking it that should help me to get over the current state, and I
> wanted to take a look at this discussion  unraveling first. Over last week
> I disconnected from "day-to-day" Airflow and put some thoughts (as much as
> I could in my current state) on it. The whole subject of this thread was
> started from that - how the current discussions on AIP-67 and others change
> if we consider Airflow 3 is "starting".
>
> The price for back-compat is speed of development and quality. More
> combinations to test, more unexpected issues uncovered, necessity to keep
> parallel paths (old/new) while adding new features. All what Constance
> wrote about and what Ash explained. We already started to trip over our own
> feet mutliple times in a few last releases. Have we tested all combinations
> of deployment in Airflow 2.8 and 2.9 - not really, I think we already see
> that in a number of "combos" of features things are not working in as
> stable a way as they did before.
>
> Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a long
> time (or even move out) as they will not want to move to Airflow 3. A lot
> of the work implemented in AIP-44 and design of AIP-67 was done around
> back-compatibility. but yes -
> it would have been way easier if designed anew without back-compatibility
> in mind. And if we implement it and release it in Airflow 2 it will make
> new Airflow feature development even harder. That's why I wanted to treat
> it as "tactical" solution - hoping that in Airflow 3 we can make it
> "properly" - and that's why I started the discussion here when I sensed
> that we are "close" to Airflow 3 discussion, because I wanted to see what
> options we have there. This is why I have not yet concluded voting on
> AIP-67 waiting for the result of this discussion here.
>
> But if we are ready to go for Airflow.3 then I'd say there are two
> important things that should be part of the vision.
>
> 1)  *We should be far more opinionated and have far fewer options of
> running things in Airflow 3*. Even an order of magnitude more opinionated.
> Make choices, stick to it, perfect those opinionated choices to suit 80/20
> (or even 70/30 or maybe even 60/40) rule if you will. Risking not fitting
> the 20% that might choose to stay at Airflow 2. We can choose now which
> ~20% of cases we do not want to handle deliberately. And we should be very,
> very strict about it. Default should be "no choice". This will radically
> simplify deployment and should make it easier to simplify Airflow
> development and DAG authoring experience because we will have less cases to
> support. Even if we plan to add more options in the future, the first
> version of Airflow 3 should support one deployment approach only. This is
> the only way we can deliver it fast. And we should be very bold there.
> Choose one option and go for it in pretty much every place we have choices
> now. We should Aim for Airflow 3.0 to support only a subset of current
> users - but those who are most likely to migrate first and those with the
> biggest need for the new features. We can think 3.x to support more cases,
> but 3.0 should be as opinionated as humanly possible.
>
> And this deployment option should be also something ALL our stakeholders
> will feel OK with as a way forward in their offering.
>
> My candidates (and yes, some are bold):
>
> * *Drop MySQL*. If we have a single thing that makes us avoid our schema
> and DB migration - this is the case. Let's choose Postgres 15+ and use some
> of the great features there. This will also enable much faster async SQL
> implementation and a number of other optimisations - not to mention cutting
> every single change in development and testing time by literally half. And
> we should not look back to adding MySQL.
> * *Drop Celery/Sequential Executor* and start with Local + K8S only (and
> AWS/Google others can continue developing theirs of course in parallel and
> continue Hybrid executor work). Later - we figure out a better solution to
> support "small" tasks using some new K8S features and possibly non-k8s
> solutions (Ray-based?)
> * *Cut Connection and Variable Management from DB/UI*. Leave only Secrets
> Management. Later when we have a 100% extensible React UI, we can add a
> "local DB secrets manager" add-on
> * *Choose a single way for DAG storage that will support versioning from
> day one*. Bear in mind we can add others later. Bolke's idea of using
> FSspec is an interesting one, we should see if it is feasible.
> * *Drop FAB completely (including custom plugins) and invest in
> implementing Auth Manager based on a dedicated, external solution*
> (KeyCloak
> that we've discussed before as a likely candidate)
> * *Leave Providers with Airflow 2 and add tests to make sure they are
> Airflow 3 future-compatible *- develop a way where we continue development
> and contributions for Providers with Airflow 2 and add complete tests to
> run them with Airflow 3. This way we can continue developing Provider
> features independently, and make them work for Airflow 2 (and continue
> adding features for Airflow 2 users alongside Airflow 2 bugfixes), while
> also gradually fix any Airflow3 incompatibilities and instead of
> "back-compatibility" tests make provider "forward-compatibility" tests so
> that future Providers are tested and work on Airflow 3. Also it will make
> it easiest to continue Airflow 2 (bugfixes) + Providers tested without
> investing in changing the current CI / test harness.
> * *Simplify Test Harness for Airflow 3 from the start *- without providers
> and 790+ dependencies, we could vastly simplify Airflow3 testing (basically
> make CI jobs from scratch) using mostly standard Python tooling (while we
> can continue making use of the current test harness for Airflow 2 +
> Providers and extend it with Airflow 3 future-compatibility tests). That
> means Breeze would be only staying in Airflow 2 + Providers repo as we
> should be able to achieve most of what we have there with local venv/
> tooling (especially with uv as underlying tooling).
>
> 2) *I think we only add very few new "important" features. *Absolute
> minimum to make Airflow 3 appealing and add them only in Airflow 3:
> versioning, multi-team, pluggable UI should only be Airflow 3 - it makes no
> sense to invest into Airflow 2 if we already know Airflow 3 is coming -
> that generally triples effort needed to get them out. We should drop new
> features development in Airflow 2. This will give users incentive to move
> to 3 if the new features will be worth it. Even paying
> compatibility/migration price.
>
> Versionig, for example: I believe if we decide to go only with Airflow 3
> and cut some of the above (Postgres only, Single versioning DAG storage) we
> can make bolder decisions in versioning and support simpler models from the
> get go (and deliver it faster). And we should add only a few - but
> important - features that our users clearly asked for and focus on
> delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or 2.11).
> Similarly - multi-team can be simplified if we cut things from the list
> above and have Task isolation as first-class citizens in Airflow (and the
> only option).
>
> My candidates very much concur with the list shared by Kaxil in the doc +
> I'd add multi-team (but simplified thanks to the cuts). But I also here
> would mostly revert to Astronomer, Google. AWS team to define collectively
> what is the absolute minimum set of features that would get the "target"
> part of their customers happy. And ONLY do that.
>
> So in short - I think the big part of our discussion should be what we are
> ready to drop when we start airflow 3 and be very bold. Once we know we
> should figure out the absolute minimum of things that we can add that will
> benefit a significant part of our users (and make use of increased speed
> because we dropped things).
>
> J.
>
>
> On Mon, May 6, 2024 at 8:40 PM Constance Martineau
> <co...@astronomer.io.invalid> wrote:
>
> > Hi Michal,
> >
> > Thanks for your thoughts on the Airflow 3 proposal. I appreciate your
> > concerns about the migration overhead for our users with a major new
> > version and see the appeal in your suggestion to integrate many of the
> > proposed changes into Airflow 2 through separate AIPs. It’s a valid point
> > and certainly aligns with the value of making incremental improvements.
> >
> > However, after looking closely at the enhancements outlined for Airflow
> 3,
> > I'm convinced they warrant a new major release. Here’s why:
> >
> >    1. *Core Architectural Changes:* We’re looking at foundational changes
> >    with Airflow 3—like redefining task priorities, separating task
> > definition
> >    and task execution, and new AIPs like DAG versioning. remote execution
> >    and restricting database access from workers. These aren’t just
> > incremental
> >    improvements but major shifts that will set the stage for the next
> > decade
> >    of Airflow’s architecture. Grouping these changes into a major release
> > will
> >    help us make these transitions more cleanly and with fewer constraints
> > from
> >    past decisions.
> >    2. *Code Clean-Up*: Our main branch has accumulated over 140
> deprecated
> >    issues, and this will only grow if we continue without a major
> cleanup.
> >    This makes it increasingly difficult to implement new features
> > effectively
> >    while maintaining backward compatibility. A major release allows us to
> >    address these issues head-on, reducing technical debt and paving the
> way
> >    for a more robust platform.
> >    3. *Managing Breaking Changes:* Let’s take the example of restricting
> >    database access from workers. It’s a necessary move for better
> security
> > and
> >    also potentially scalability reasons (reduces DB load). Many users
> have
> >    workflows that interact with the DB, either by using raw sql or by
> >    leveraging a session object. We could implement this feature in
> Airflow
> > 2
> >    and avoid breaking existing workflows by continuing to have the old
> >    standard mode as default - much of the work is already done - but that
> >    would mean supporting both the new secure mode and the old standard
> mode
> >    indefinitely and design new features with the assumption that most
> will
> >    continue using the old standard mode. With Airflow 3, we can make
> secure
> >    mode the default or even the only option, simplifying implementation
> and
> >    future development. This is just one example where it is feasible to
> >    implement in Airflow 2, but is better if we release it under the
> > context of
> >    Airflow 3.
> >    4. *Future-Proofing for New Features:* Airflow 3 will open up
> >    possibilities for handling workflows beyond batch processing. Features
> > like
> >    real-time DAG execution through API and multi-language task support
> are
> > big
> >    steps forward, significantly expanding Airflow’s utility.
> >
> >
> > While integrating these updates into Airflow 2 might look less disruptive
> > initially, the scale and nature of the required changes really support a
> > move to Airflow 3. It’s not just about adding new features; it’s about
> > setting up Airflow so that it continues to remain relevant for the next
> ten
> > years.
> >
> > Constance
> >
> > On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > > There's a lot of technical debt hiding in Airflow, especially the
> > > scheduler that makes it harder and harder to efficiently add new
> > features.
> > >
> > > At some point, very soon, we are going to have to remove some very
> > > infrequently used back compat shims that negatively affect performance.
> > > Without doing that the pace at which we can realistically add some of
> the
> > > more exciting features tends towards zero. Developer speed of
> > contributors
> > > is a factor here too!
> > >
> > > So while we are still using SemVer, that necessitates v3.
> > >
> > > Ash
> > >
> > > On 6 May 2024 15:30:49 BST, "Michał Modras" <michalmodras@google.com
> > .INVALID>
> > > wrote:
> > > >+1 to Jens's & Bolke's points here and in the doc
> > > >
> > > >I agree we should work on clarifying the directions we would like
> > Airflow
> > > >to go. Introducing a new major Airflow version is a massive overhead
> for
> > > >users, who would need to plan for migrations, onboarding the new
> Airflow
> > > >(with a slightly different architecture), etc., and effectively
> Airflow
> > 2
> > > >would live in parallel for a long time.
> > > >
> > > >Personally, I think most of the points in Kaxil's/Vikram's doc are
> > > valuable
> > > >projects of their own, and I could imagine all of them being delivered
> > as
> > > >separate AIPs within Airflow 2 (surely new minor versions of Airflow
> > 2). I
> > > >am not sure if the scope of changes and the goal we want to achieve is
> > a)
> > > >clear enough b) broad enough to call for a new major version.
> > > >
> > > >Best,
> > > >Michal
> > > >
> > > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> > > ><Je...@de.bosch.com.invalid> wrote:
> > > >
> > > >> Thanks for the document write-up, Kaxil. I assume this is mostly a
> > > vision
> > > >> statement.
> > > >>
> > > >> Looking forward for a larger addendum where we can collect things
> that
> > > we
> > > >> all can vote and agree on as targets.
> > > >>
> > > >> As I started earlier with a confluence page and it seems this is not
> > > >> accessible to all, shall we convert this to a Google Doc for better
> > > >> collaboration and item collection?
> > > >>
> > > >> Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > > >> ________________________________
> > > >> From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> > > >> To: dev@airflow.apache.org <de...@airflow.apache.org>
> > > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
> > > >> strategic (Airflow 3) approach
> > > >>
> > > >> Thank you for your feedback, Bolke and Andrey!
> > > >>
> > > >> Bolke,
> > > >> I have replied to some of your comments in the doc.
> > > >> I will provide a detailed write up on the "Interactive DAG run" (or
> > > >> synchronous DAG run) capability, which has generated some early
> > > questions.
> > > >> I had intended to get an AIP published for that as a follow-up, but
> I
> > > >> believe that a simpler write up would be useful ahead of the AIP.
> > > >>
> > > >> Andrey,
> > > >> You raise an interesting point.
> > > >>
> > > >> As part of the Airflow 2.0 release, we as a community had decided to
> > > >> strictly adhere to Semver as detailed in the document you
> referenced.
> > We
> > > >> also consciously split out the "Core Airflow" releases from the
> > > "Provider"
> > > >> releases at that time. We had a clear expectation then for the
> cadence
> > > of
> > > >> both minor and patch releases, which we have generally adhered to
> > since
> > > >> then.
> > > >>
> > > >> Personally, I am more concerned about our Provider releases right
> now,
> > > as
> > > >> compared to the cadence of our major releases. I believe that one of
> > the
> > > >> proposed changes in the Airflow 3 document i.e. the clear separation
> > for
> > > >> Task Execution will help here, but more may be needed.
> > > >>
> > > >> Definitely interested in more feedback on this as well.
> > > >>
> > > >> Vikram
> > > >>
> > > >>
> > > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> > andrey.anshin@taragol.is
> > > >
> > > >> wrote:
> > > >>
> > > >> > I would like to propose to change (at least discuss) release
> policy
> > > >> around
> > > >> > the Major version of Airflow.
> > > >> >
> > > >> > Right now it is described as "These releases do not happen with
> any
> > > >> regular
> > > >> > interval or on any predictable schedule." :
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> > > >> <
> > > >>
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > > >> >
> > > >> >
> > > >> > So maybe it is time to make it schedulable, e.g. one per two years
> > or
> > > so.
> > > >> > This one could help us to avoid such a discussion in the future,
> > like
> > > "We
> > > >> > don't know when Airflow 4 is coming.". At the moment when the new
> > > major
> > > >> > version will be released new features wouldn't be added in the old
> > > major
> > > >> > version, however we would support bug / security for a while,
> e.g. 1
> > > year
> > > >> > for bug fixes, 3 years for security fixes with a total 5 year
> > > lifecycle
> > > >> per
> > > >> > a major version. These just are approximate time periods for a
> > > definition
> > > >> > of current period, bugfix period and security fix period.
> > > >> >
> > > >> > In contributors' perspective it helps with dropping the deprecated
> > > stuff
> > > >> > which resolves some old problem: we have to support everything
> > > including
> > > >> > deprecated stuff and without schedulable lifecycle for the
> > deprecated
> > > >> stuff
> > > >> > it could be showstopper for the new feature, because sometimes it
> > > hard to
> > > >> > support two different approaches for long period of time with no
> > hope
> > > >> that
> > > >> > it will happen soon. For some fundamental stuff which do not
> > require a
> > > >> lot
> > > >> > things time to support we could postponed removal for next after
> the
> > > next
> > > >> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
> > > >> >
> > > >> > In the user perspective, they have at least bug fix support for a
> > > while,
> > > >> if
> > > >> > someone want to use legacy version it their choice, however no new
> > > >> > features, no new version of providers (after one year)
> > > >> >
> > > >> >
> > > >> > ----
> > > >> > Best Wishes
> > > >> > *Andrey Anshin*
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bd...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > I have left several comments :-). And on interactive dag runs
> even
> > > >> after
> > > >> > > the explanation of Vikram I still don't have a clue what we want
> > to
> > > >> > > accomplish there :-P.
> > > >> > >
> > > >> > > I would like to see a mantra or team for Airflow 3. That helps
> > > nudging
> > > >> > > people in the same direction. Suggestions in the comments.
> > > >> > >
> > > >> > > Bolke
> > > >> > > Sent from my iPhone
> > > >> > >
> > > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> > <vikram@astronomer.io.invalid
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > Good point Jed.
> > > >> > > > I responded back to your comment in the doc as well and very
> > open
> > > to
> > > >> > > > changing the term in the doc.
> > > >> > > >
> > > >> > > > Used the term "interactive DAG run" as the ability to invoke
> or
> > > >> > trigger a
> > > >> > > > DAG run through the API, with the expectation of getting back
> a
> > > >> result
> > > >> > > > immediately. An alternate term could be a "synchronous DAG
> run".
> > > >> > > >
> > > >> > > > Regardless, this is a significant change so a good term to
> > > indicate
> > > >> the
> > > >> > > > expansion from "batch runs only" is warranted. Very open to
> > > different
> > > >> > > terms
> > > >> > > > here.
> > > >> > > >
> > > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > > >> > jedcunningham@apache.org
> > > >> > > >
> > > >> > > >> wrote:
> > > >> > > >>
> > > >> > > >> Very exciting! Looks like we will have a busy period of time
> > > ahead
> > > >> of
> > > >> > > us.
> > > >> > > >> Overall I like the plan so far, especially using this year's
> > > Airflow
> > > >> > > Summit
> > > >> > > >> as an opportunity to announce and gather feedback, and the
> 2025
> > > >> > version
> > > >> > > to
> > > >> > > >> pitch upgrading.
> > > >> > > >>
> > > >> > > >> I left a comment in the doc, but we might want to iterate on
> > the
> > > >> > > >> terminology we use for high priority or "synchronous" DAG
> runs
> > to
> > > >> > serve
> > > >> > > LLM
> > > >> > > >> responses - I find "interactive DAG runs" a bit confusing.
> > > >> > > >>
> > > >> > >
> > > >> > >
> > > ---------------------------------------------------------------------
> > > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Jarek Potiuk <ja...@potiuk.com>]

I am currently on sick leave, and still recovering - hoping to be able to
travel next week to the US as planned, so I just wanted to break out of it
to make one comment here.

I got a clearer head now a bit with medications hopefully working. I am
still taking it that should help me to get over the current state, and I
wanted to take a look at this discussion  unraveling first. Over last week
I disconnected from "day-to-day" Airflow and put some thoughts (as much as
I could in my current state) on it. The whole subject of this thread was
started from that - how the current discussions on AIP-67 and others change
if we consider Airflow 3 is "starting".

The price for back-compat is speed of development and quality. More
combinations to test, more unexpected issues uncovered, necessity to keep
parallel paths (old/new) while adding new features. All what Constance
wrote about and what Ash explained. We already started to trip over our own
feet mutliple times in a few last releases. Have we tested all combinations
of deployment in Airflow 2.8 and 2.9 - not really, I think we already see
that in a number of "combos" of features things are not working in as
stable a way as they did before.

Airflow 3 is a bold move. We risk users will stay on Airflow 2 for a long
time (or even move out) as they will not want to move to Airflow 3. A lot
of the work implemented in AIP-44 and design of AIP-67 was done around
back-compatibility. but yes -
it would have been way easier if designed anew without back-compatibility
in mind. And if we implement it and release it in Airflow 2 it will make
new Airflow feature development even harder. That's why I wanted to treat
it as "tactical" solution - hoping that in Airflow 3 we can make it
"properly" - and that's why I started the discussion here when I sensed
that we are "close" to Airflow 3 discussion, because I wanted to see what
options we have there. This is why I have not yet concluded voting on
AIP-67 waiting for the result of this discussion here.

But if we are ready to go for Airflow.3 then I'd say there are two
important things that should be part of the vision.

1)  *We should be far more opinionated and have far fewer options of
running things in Airflow 3*. Even an order of magnitude more opinionated.
Make choices, stick to it, perfect those opinionated choices to suit 80/20
(or even 70/30 or maybe even 60/40) rule if you will. Risking not fitting
the 20% that might choose to stay at Airflow 2. We can choose now which
~20% of cases we do not want to handle deliberately. And we should be very,
very strict about it. Default should be "no choice". This will radically
simplify deployment and should make it easier to simplify Airflow
development and DAG authoring experience because we will have less cases to
support. Even if we plan to add more options in the future, the first
version of Airflow 3 should support one deployment approach only. This is
the only way we can deliver it fast. And we should be very bold there.
Choose one option and go for it in pretty much every place we have choices
now. We should Aim for Airflow 3.0 to support only a subset of current
users - but those who are most likely to migrate first and those with the
biggest need for the new features. We can think 3.x to support more cases,
but 3.0 should be as opinionated as humanly possible.

And this deployment option should be also something ALL our stakeholders
will feel OK with as a way forward in their offering.

My candidates (and yes, some are bold):

* *Drop MySQL*. If we have a single thing that makes us avoid our schema
and DB migration - this is the case. Let's choose Postgres 15+ and use some
of the great features there. This will also enable much faster async SQL
implementation and a number of other optimisations - not to mention cutting
every single change in development and testing time by literally half. And
we should not look back to adding MySQL.
* *Drop Celery/Sequential Executor* and start with Local + K8S only (and
AWS/Google others can continue developing theirs of course in parallel and
continue Hybrid executor work). Later - we figure out a better solution to
support "small" tasks using some new K8S features and possibly non-k8s
solutions (Ray-based?)
* *Cut Connection and Variable Management from DB/UI*. Leave only Secrets
Management. Later when we have a 100% extensible React UI, we can add a
"local DB secrets manager" add-on
* *Choose a single way for DAG storage that will support versioning from
day one*. Bear in mind we can add others later. Bolke's idea of using
FSspec is an interesting one, we should see if it is feasible.
* *Drop FAB completely (including custom plugins) and invest in
implementing Auth Manager based on a dedicated, external solution* (KeyCloak
that we've discussed before as a likely candidate)
* *Leave Providers with Airflow 2 and add tests to make sure they are
Airflow 3 future-compatible *- develop a way where we continue development
and contributions for Providers with Airflow 2 and add complete tests to
run them with Airflow 3. This way we can continue developing Provider
features independently, and make them work for Airflow 2 (and continue
adding features for Airflow 2 users alongside Airflow 2 bugfixes), while
also gradually fix any Airflow3 incompatibilities and instead of
"back-compatibility" tests make provider "forward-compatibility" tests so
that future Providers are tested and work on Airflow 3. Also it will make
it easiest to continue Airflow 2 (bugfixes) + Providers tested without
investing in changing the current CI / test harness.
* *Simplify Test Harness for Airflow 3 from the start *- without providers
and 790+ dependencies, we could vastly simplify Airflow3 testing (basically
make CI jobs from scratch) using mostly standard Python tooling (while we
can continue making use of the current test harness for Airflow 2 +
Providers and extend it with Airflow 3 future-compatibility tests). That
means Breeze would be only staying in Airflow 2 + Providers repo as we
should be able to achieve most of what we have there with local venv/
tooling (especially with uv as underlying tooling).

2) *I think we only add very few new "important" features. *Absolute
minimum to make Airflow 3 appealing and add them only in Airflow 3:
versioning, multi-team, pluggable UI should only be Airflow 3 - it makes no
sense to invest into Airflow 2 if we already know Airflow 3 is coming -
that generally triples effort needed to get them out. We should drop new
features development in Airflow 2. This will give users incentive to move
to 3 if the new features will be worth it. Even paying
compatibility/migration price.

Versionig, for example: I believe if we decide to go only with Airflow 3
and cut some of the above (Postgres only, Single versioning DAG storage) we
can make bolder decisions in versioning and support simpler models from the
get go (and deliver it faster). And we should add only a few - but
important - features that our users clearly asked for and focus on
delivering Airflow 3 as soon as possible (instead of Airflow 2.10 or 2.11).
Similarly - multi-team can be simplified if we cut things from the list
above and have Task isolation as first-class citizens in Airflow (and the
only option).

My candidates very much concur with the list shared by Kaxil in the doc +
I'd add multi-team (but simplified thanks to the cuts). But I also here
would mostly revert to Astronomer, Google. AWS team to define collectively
what is the absolute minimum set of features that would get the "target"
part of their customers happy. And ONLY do that.

So in short - I think the big part of our discussion should be what we are
ready to drop when we start airflow 3 and be very bold. Once we know we
should figure out the absolute minimum of things that we can add that will
benefit a significant part of our users (and make use of increased speed
because we dropped things).

J.


On Mon, May 6, 2024 at 8:40 PM Constance Martineau
<co...@astronomer.io.invalid> wrote:

> Hi Michal,
>
> Thanks for your thoughts on the Airflow 3 proposal. I appreciate your
> concerns about the migration overhead for our users with a major new
> version and see the appeal in your suggestion to integrate many of the
> proposed changes into Airflow 2 through separate AIPs. It’s a valid point
> and certainly aligns with the value of making incremental improvements.
>
> However, after looking closely at the enhancements outlined for Airflow 3,
> I'm convinced they warrant a new major release. Here’s why:
>
>    1. *Core Architectural Changes:* We’re looking at foundational changes
>    with Airflow 3—like redefining task priorities, separating task
> definition
>    and task execution, and new AIPs like DAG versioning. remote execution
>    and restricting database access from workers. These aren’t just
> incremental
>    improvements but major shifts that will set the stage for the next
> decade
>    of Airflow’s architecture. Grouping these changes into a major release
> will
>    help us make these transitions more cleanly and with fewer constraints
> from
>    past decisions.
>    2. *Code Clean-Up*: Our main branch has accumulated over 140 deprecated
>    issues, and this will only grow if we continue without a major cleanup.
>    This makes it increasingly difficult to implement new features
> effectively
>    while maintaining backward compatibility. A major release allows us to
>    address these issues head-on, reducing technical debt and paving the way
>    for a more robust platform.
>    3. *Managing Breaking Changes:* Let’s take the example of restricting
>    database access from workers. It’s a necessary move for better security
> and
>    also potentially scalability reasons (reduces DB load). Many users have
>    workflows that interact with the DB, either by using raw sql or by
>    leveraging a session object. We could implement this feature in Airflow
> 2
>    and avoid breaking existing workflows by continuing to have the old
>    standard mode as default - much of the work is already done - but that
>    would mean supporting both the new secure mode and the old standard mode
>    indefinitely and design new features with the assumption that most will
>    continue using the old standard mode. With Airflow 3, we can make secure
>    mode the default or even the only option, simplifying implementation and
>    future development. This is just one example where it is feasible to
>    implement in Airflow 2, but is better if we release it under the
> context of
>    Airflow 3.
>    4. *Future-Proofing for New Features:* Airflow 3 will open up
>    possibilities for handling workflows beyond batch processing. Features
> like
>    real-time DAG execution through API and multi-language task support are
> big
>    steps forward, significantly expanding Airflow’s utility.
>
>
> While integrating these updates into Airflow 2 might look less disruptive
> initially, the scale and nature of the required changes really support a
> move to Airflow 3. It’s not just about adding new features; it’s about
> setting up Airflow so that it continues to remain relevant for the next ten
> years.
>
> Constance
>
> On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > There's a lot of technical debt hiding in Airflow, especially the
> > scheduler that makes it harder and harder to efficiently add new
> features.
> >
> > At some point, very soon, we are going to have to remove some very
> > infrequently used back compat shims that negatively affect performance.
> > Without doing that the pace at which we can realistically add some of the
> > more exciting features tends towards zero. Developer speed of
> contributors
> > is a factor here too!
> >
> > So while we are still using SemVer, that necessitates v3.
> >
> > Ash
> >
> > On 6 May 2024 15:30:49 BST, "Michał Modras" <michalmodras@google.com
> .INVALID>
> > wrote:
> > >+1 to Jens's & Bolke's points here and in the doc
> > >
> > >I agree we should work on clarifying the directions we would like
> Airflow
> > >to go. Introducing a new major Airflow version is a massive overhead for
> > >users, who would need to plan for migrations, onboarding the new Airflow
> > >(with a slightly different architecture), etc., and effectively Airflow
> 2
> > >would live in parallel for a long time.
> > >
> > >Personally, I think most of the points in Kaxil's/Vikram's doc are
> > valuable
> > >projects of their own, and I could imagine all of them being delivered
> as
> > >separate AIPs within Airflow 2 (surely new minor versions of Airflow
> 2). I
> > >am not sure if the scope of changes and the goal we want to achieve is
> a)
> > >clear enough b) broad enough to call for a new major version.
> > >
> > >Best,
> > >Michal
> > >
> > >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> > ><Je...@de.bosch.com.invalid> wrote:
> > >
> > >> Thanks for the document write-up, Kaxil. I assume this is mostly a
> > vision
> > >> statement.
> > >>
> > >> Looking forward for a larger addendum where we can collect things that
> > we
> > >> all can vote and agree on as targets.
> > >>
> > >> As I started earlier with a confluence page and it seems this is not
> > >> accessible to all, shall we convert this to a Google Doc for better
> > >> collaboration and item collection?
> > >>
> > >> Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > >> ________________________________
> > >> From: Vikram Koka <vi...@astronomer.io.INVALID>
> > >> Sent: Sunday, May 5, 2024 3:34:33 AM
> > >> To: dev@airflow.apache.org <de...@airflow.apache.org>
> > >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
> > >> strategic (Airflow 3) approach
> > >>
> > >> Thank you for your feedback, Bolke and Andrey!
> > >>
> > >> Bolke,
> > >> I have replied to some of your comments in the doc.
> > >> I will provide a detailed write up on the "Interactive DAG run" (or
> > >> synchronous DAG run) capability, which has generated some early
> > questions.
> > >> I had intended to get an AIP published for that as a follow-up, but I
> > >> believe that a simpler write up would be useful ahead of the AIP.
> > >>
> > >> Andrey,
> > >> You raise an interesting point.
> > >>
> > >> As part of the Airflow 2.0 release, we as a community had decided to
> > >> strictly adhere to Semver as detailed in the document you referenced.
> We
> > >> also consciously split out the "Core Airflow" releases from the
> > "Provider"
> > >> releases at that time. We had a clear expectation then for the cadence
> > of
> > >> both minor and patch releases, which we have generally adhered to
> since
> > >> then.
> > >>
> > >> Personally, I am more concerned about our Provider releases right now,
> > as
> > >> compared to the cadence of our major releases. I believe that one of
> the
> > >> proposed changes in the Airflow 3 document i.e. the clear separation
> for
> > >> Task Execution will help here, but more may be needed.
> > >>
> > >> Definitely interested in more feedback on this as well.
> > >>
> > >> Vikram
> > >>
> > >>
> > >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <
> andrey.anshin@taragol.is
> > >
> > >> wrote:
> > >>
> > >> > I would like to propose to change (at least discuss) release policy
> > >> around
> > >> > the Major version of Airflow.
> > >> >
> > >> > Right now it is described as "These releases do not happen with any
> > >> regular
> > >> > interval or on any predictable schedule." :
> > >> >
> > >> >
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> > >> <
> > >>
> >
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> > >> >
> > >> >
> > >> > So maybe it is time to make it schedulable, e.g. one per two years
> or
> > so.
> > >> > This one could help us to avoid such a discussion in the future,
> like
> > "We
> > >> > don't know when Airflow 4 is coming.". At the moment when the new
> > major
> > >> > version will be released new features wouldn't be added in the old
> > major
> > >> > version, however we would support bug / security for a while, e.g. 1
> > year
> > >> > for bug fixes, 3 years for security fixes with a total 5 year
> > lifecycle
> > >> per
> > >> > a major version. These just are approximate time periods for a
> > definition
> > >> > of current period, bugfix period and security fix period.
> > >> >
> > >> > In contributors' perspective it helps with dropping the deprecated
> > stuff
> > >> > which resolves some old problem: we have to support everything
> > including
> > >> > deprecated stuff and without schedulable lifecycle for the
> deprecated
> > >> stuff
> > >> > it could be showstopper for the new feature, because sometimes it
> > hard to
> > >> > support two different approaches for long period of time with no
> hope
> > >> that
> > >> > it will happen soon. For some fundamental stuff which do not
> require a
> > >> lot
> > >> > things time to support we could postponed removal for next after the
> > next
> > >> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
> > >> >
> > >> > In the user perspective, they have at least bug fix support for a
> > while,
> > >> if
> > >> > someone want to use legacy version it their choice, however no new
> > >> > features, no new version of providers (after one year)
> > >> >
> > >> >
> > >> > ----
> > >> > Best Wishes
> > >> > *Andrey Anshin*
> > >> >
> > >> >
> > >> >
> > >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >> >
> > >> > > I have left several comments :-). And on interactive dag runs even
> > >> after
> > >> > > the explanation of Vikram I still don't have a clue what we want
> to
> > >> > > accomplish there :-P.
> > >> > >
> > >> > > I would like to see a mantra or team for Airflow 3. That helps
> > nudging
> > >> > > people in the same direction. Suggestions in the comments.
> > >> > >
> > >> > > Bolke
> > >> > > Sent from my iPhone
> > >> > >
> > >> > > > On 4 May 2024, at 01:14, Vikram Koka
> <vikram@astronomer.io.invalid
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > Good point Jed.
> > >> > > > I responded back to your comment in the doc as well and very
> open
> > to
> > >> > > > changing the term in the doc.
> > >> > > >
> > >> > > > Used the term "interactive DAG run" as the ability to invoke or
> > >> > trigger a
> > >> > > > DAG run through the API, with the expectation of getting back a
> > >> result
> > >> > > > immediately. An alternate term could be a "synchronous DAG run".
> > >> > > >
> > >> > > > Regardless, this is a significant change so a good term to
> > indicate
> > >> the
> > >> > > > expansion from "batch runs only" is warranted. Very open to
> > different
> > >> > > terms
> > >> > > > here.
> > >> > > >
> > >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > >> > jedcunningham@apache.org
> > >> > > >
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> Very exciting! Looks like we will have a busy period of time
> > ahead
> > >> of
> > >> > > us.
> > >> > > >> Overall I like the plan so far, especially using this year's
> > Airflow
> > >> > > Summit
> > >> > > >> as an opportunity to announce and gather feedback, and the 2025
> > >> > version
> > >> > > to
> > >> > > >> pitch upgrading.
> > >> > > >>
> > >> > > >> I left a comment in the doc, but we might want to iterate on
> the
> > >> > > >> terminology we use for high priority or "synchronous" DAG runs
> to
> > >> > serve
> > >> > > LLM
> > >> > > >> responses - I find "interactive DAG runs" a bit confusing.
> > >> > > >>
> > >> > >
> > >> > >
> > ---------------------------------------------------------------------
> > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Constance Martineau <co...@astronomer.io.INVALID>]

Hi Michal,

Thanks for your thoughts on the Airflow 3 proposal. I appreciate your
concerns about the migration overhead for our users with a major new
version and see the appeal in your suggestion to integrate many of the
proposed changes into Airflow 2 through separate AIPs. It’s a valid point
and certainly aligns with the value of making incremental improvements.

However, after looking closely at the enhancements outlined for Airflow 3,
I'm convinced they warrant a new major release. Here’s why:

   1. *Core Architectural Changes:* We’re looking at foundational changes
   with Airflow 3—like redefining task priorities, separating task definition
   and task execution, and new AIPs like DAG versioning. remote execution
   and restricting database access from workers. These aren’t just incremental
   improvements but major shifts that will set the stage for the next decade
   of Airflow’s architecture. Grouping these changes into a major release will
   help us make these transitions more cleanly and with fewer constraints from
   past decisions.
   2. *Code Clean-Up*: Our main branch has accumulated over 140 deprecated
   issues, and this will only grow if we continue without a major cleanup.
   This makes it increasingly difficult to implement new features effectively
   while maintaining backward compatibility. A major release allows us to
   address these issues head-on, reducing technical debt and paving the way
   for a more robust platform.
   3. *Managing Breaking Changes:* Let’s take the example of restricting
   database access from workers. It’s a necessary move for better security and
   also potentially scalability reasons (reduces DB load). Many users have
   workflows that interact with the DB, either by using raw sql or by
   leveraging a session object. We could implement this feature in Airflow 2
   and avoid breaking existing workflows by continuing to have the old
   standard mode as default - much of the work is already done - but that
   would mean supporting both the new secure mode and the old standard mode
   indefinitely and design new features with the assumption that most will
   continue using the old standard mode. With Airflow 3, we can make secure
   mode the default or even the only option, simplifying implementation and
   future development. This is just one example where it is feasible to
   implement in Airflow 2, but is better if we release it under the context of
   Airflow 3.
   4. *Future-Proofing for New Features:* Airflow 3 will open up
   possibilities for handling workflows beyond batch processing. Features like
   real-time DAG execution through API and multi-language task support are big
   steps forward, significantly expanding Airflow’s utility.


While integrating these updates into Airflow 2 might look less disruptive
initially, the scale and nature of the required changes really support a
move to Airflow 3. It’s not just about adding new features; it’s about
setting up Airflow so that it continues to remain relevant for the next ten
years.

Constance

On Mon, May 6, 2024 at 2:10 PM Ash Berlin-Taylor <as...@apache.org> wrote:

> There's a lot of technical debt hiding in Airflow, especially the
> scheduler that makes it harder and harder to efficiently add new features.
>
> At some point, very soon, we are going to have to remove some very
> infrequently used back compat shims that negatively affect performance.
> Without doing that the pace at which we can realistically add some of the
> more exciting features tends towards zero. Developer speed of contributors
> is a factor here too!
>
> So while we are still using SemVer, that necessitates v3.
>
> Ash
>
> On 6 May 2024 15:30:49 BST, "Michał Modras" <mi...@google.com.INVALID>
> wrote:
> >+1 to Jens's & Bolke's points here and in the doc
> >
> >I agree we should work on clarifying the directions we would like Airflow
> >to go. Introducing a new major Airflow version is a massive overhead for
> >users, who would need to plan for migrations, onboarding the new Airflow
> >(with a slightly different architecture), etc., and effectively Airflow 2
> >would live in parallel for a long time.
> >
> >Personally, I think most of the points in Kaxil's/Vikram's doc are
> valuable
> >projects of their own, and I could imagine all of them being delivered as
> >separate AIPs within Airflow 2 (surely new minor versions of Airflow 2). I
> >am not sure if the scope of changes and the goal we want to achieve is a)
> >clear enough b) broad enough to call for a new major version.
> >
> >Best,
> >Michal
> >
> >On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T)
> ><Je...@de.bosch.com.invalid> wrote:
> >
> >> Thanks for the document write-up, Kaxil. I assume this is mostly a
> vision
> >> statement.
> >>
> >> Looking forward for a larger addendum where we can collect things that
> we
> >> all can vote and agree on as targets.
> >>
> >> As I started earlier with a confluence page and it seems this is not
> >> accessible to all, shall we convert this to a Google Doc for better
> >> collaboration and item collection?
> >>
> >> Sent from Outlook for iOS<https://aka.ms/o0ukef>
> >> ________________________________
> >> From: Vikram Koka <vi...@astronomer.io.INVALID>
> >> Sent: Sunday, May 5, 2024 3:34:33 AM
> >> To: dev@airflow.apache.org <de...@airflow.apache.org>
> >> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
> >> strategic (Airflow 3) approach
> >>
> >> Thank you for your feedback, Bolke and Andrey!
> >>
> >> Bolke,
> >> I have replied to some of your comments in the doc.
> >> I will provide a detailed write up on the "Interactive DAG run" (or
> >> synchronous DAG run) capability, which has generated some early
> questions.
> >> I had intended to get an AIP published for that as a follow-up, but I
> >> believe that a simpler write up would be useful ahead of the AIP.
> >>
> >> Andrey,
> >> You raise an interesting point.
> >>
> >> As part of the Airflow 2.0 release, we as a community had decided to
> >> strictly adhere to Semver as detailed in the document you referenced. We
> >> also consciously split out the "Core Airflow" releases from the
> "Provider"
> >> releases at that time. We had a clear expectation then for the cadence
> of
> >> both minor and patch releases, which we have generally adhered to since
> >> then.
> >>
> >> Personally, I am more concerned about our Provider releases right now,
> as
> >> compared to the cadence of our major releases. I believe that one of the
> >> proposed changes in the Airflow 3 document i.e. the clear separation for
> >> Task Execution will help here, but more may be needed.
> >>
> >> Definitely interested in more feedback on this as well.
> >>
> >> Vikram
> >>
> >>
> >> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <andrey.anshin@taragol.is
> >
> >> wrote:
> >>
> >> > I would like to propose to change (at least discuss) release policy
> >> around
> >> > the Major version of Airflow.
> >> >
> >> > Right now it is described as "These releases do not happen with any
> >> regular
> >> > interval or on any predictable schedule." :
> >> >
> >> >
> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> >> <
> >>
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> >> >
> >> >
> >> > So maybe it is time to make it schedulable, e.g. one per two years or
> so.
> >> > This one could help us to avoid such a discussion in the future, like
> "We
> >> > don't know when Airflow 4 is coming.". At the moment when the new
> major
> >> > version will be released new features wouldn't be added in the old
> major
> >> > version, however we would support bug / security for a while, e.g. 1
> year
> >> > for bug fixes, 3 years for security fixes with a total 5 year
> lifecycle
> >> per
> >> > a major version. These just are approximate time periods for a
> definition
> >> > of current period, bugfix period and security fix period.
> >> >
> >> > In contributors' perspective it helps with dropping the deprecated
> stuff
> >> > which resolves some old problem: we have to support everything
> including
> >> > deprecated stuff and without schedulable lifecycle for the deprecated
> >> stuff
> >> > it could be showstopper for the new feature, because sometimes it
> hard to
> >> > support two different approaches for long period of time with no hope
> >> that
> >> > it will happen soon. For some fundamental stuff which do not require a
> >> lot
> >> > things time to support we could postponed removal for next after the
> next
> >> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
> >> >
> >> > In the user perspective, they have at least bug fix support for a
> while,
> >> if
> >> > someone want to use legacy version it their choice, however no new
> >> > features, no new version of providers (after one year)
> >> >
> >> >
> >> > ----
> >> > Best Wishes
> >> > *Andrey Anshin*
> >> >
> >> >
> >> >
> >> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >> >
> >> > > I have left several comments :-). And on interactive dag runs even
> >> after
> >> > > the explanation of Vikram I still don't have a clue what we want to
> >> > > accomplish there :-P.
> >> > >
> >> > > I would like to see a mantra or team for Airflow 3. That helps
> nudging
> >> > > people in the same direction. Suggestions in the comments.
> >> > >
> >> > > Bolke
> >> > > Sent from my iPhone
> >> > >
> >> > > > On 4 May 2024, at 01:14, Vikram Koka <vikram@astronomer.io.invalid
> >
> >> > > wrote:
> >> > > >
> >> > > > Good point Jed.
> >> > > > I responded back to your comment in the doc as well and very open
> to
> >> > > > changing the term in the doc.
> >> > > >
> >> > > > Used the term "interactive DAG run" as the ability to invoke or
> >> > trigger a
> >> > > > DAG run through the API, with the expectation of getting back a
> >> result
> >> > > > immediately. An alternate term could be a "synchronous DAG run".
> >> > > >
> >> > > > Regardless, this is a significant change so a good term to
> indicate
> >> the
> >> > > > expansion from "batch runs only" is warranted. Very open to
> different
> >> > > terms
> >> > > > here.
> >> > > >
> >> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> >> > jedcunningham@apache.org
> >> > > >
> >> > > >> wrote:
> >> > > >>
> >> > > >> Very exciting! Looks like we will have a busy period of time
> ahead
> >> of
> >> > > us.
> >> > > >> Overall I like the plan so far, especially using this year's
> Airflow
> >> > > Summit
> >> > > >> as an opportunity to announce and gather feedback, and the 2025
> >> > version
> >> > > to
> >> > > >> pitch upgrading.
> >> > > >>
> >> > > >> I left a comment in the doc, but we might want to iterate on the
> >> > > >> terminology we use for high priority or "synchronous" DAG runs to
> >> > serve
> >> > > LLM
> >> > > >> responses - I find "interactive DAG runs" a bit confusing.
> >> > > >>
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> >> > >
> >> > >
> >> >
> >>
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Ash Berlin-Taylor <as...@apache.org>]

There's a lot of technical debt hiding in Airflow, especially the scheduler that makes it harder and harder to efficiently add new features.

At some point, very soon, we are going to have to remove some very infrequently used back compat shims that negatively affect performance. Without doing that the pace at which we can realistically add some of the more exciting features tends towards zero. Developer speed of contributors is a factor here too!

So while we are still using SemVer, that necessitates v3.

Ash 

On 6 May 2024 15:30:49 BST, "Michał Modras" <mi...@google.com.INVALID> wrote:
>+1 to Jens's & Bolke's points here and in the doc
>
>I agree we should work on clarifying the directions we would like Airflow
>to go. Introducing a new major Airflow version is a massive overhead for
>users, who would need to plan for migrations, onboarding the new Airflow
>(with a slightly different architecture), etc., and effectively Airflow 2
>would live in parallel for a long time.
>
>Personally, I think most of the points in Kaxil's/Vikram's doc are valuable
>projects of their own, and I could imagine all of them being delivered as
>separate AIPs within Airflow 2 (surely new minor versions of Airflow 2). I
>am not sure if the scope of changes and the goal we want to achieve is a)
>clear enough b) broad enough to call for a new major version.
>
>Best,
>Michal
>
>On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T)
><Je...@de.bosch.com.invalid> wrote:
>
>> Thanks for the document write-up, Kaxil. I assume this is mostly a vision
>> statement.
>>
>> Looking forward for a larger addendum where we can collect things that we
>> all can vote and agree on as targets.
>>
>> As I started earlier with a confluence page and it seems this is not
>> accessible to all, shall we convert this to a Google Doc for better
>> collaboration and item collection?
>>
>> Sent from Outlook for iOS<https://aka.ms/o0ukef>
>> ________________________________
>> From: Vikram Koka <vi...@astronomer.io.INVALID>
>> Sent: Sunday, May 5, 2024 3:34:33 AM
>> To: dev@airflow.apache.org <de...@airflow.apache.org>
>> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
>> strategic (Airflow 3) approach
>>
>> Thank you for your feedback, Bolke and Andrey!
>>
>> Bolke,
>> I have replied to some of your comments in the doc.
>> I will provide a detailed write up on the "Interactive DAG run" (or
>> synchronous DAG run) capability, which has generated some early questions.
>> I had intended to get an AIP published for that as a follow-up, but I
>> believe that a simpler write up would be useful ahead of the AIP.
>>
>> Andrey,
>> You raise an interesting point.
>>
>> As part of the Airflow 2.0 release, we as a community had decided to
>> strictly adhere to Semver as detailed in the document you referenced. We
>> also consciously split out the "Core Airflow" releases from the "Provider"
>> releases at that time. We had a clear expectation then for the cadence of
>> both minor and patch releases, which we have generally adhered to since
>> then.
>>
>> Personally, I am more concerned about our Provider releases right now, as
>> compared to the cadence of our major releases. I believe that one of the
>> proposed changes in the Airflow 3 document i.e. the clear separation for
>> Task Execution will help here, but more may be needed.
>>
>> Definitely interested in more feedback on this as well.
>>
>> Vikram
>>
>>
>> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <an...@taragol.is>
>> wrote:
>>
>> > I would like to propose to change (at least discuss) release policy
>> around
>> > the Major version of Airflow.
>> >
>> > Right now it is described as "These releases do not happen with any
>> regular
>> > interval or on any predictable schedule." :
>> >
>> >
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
>> <
>> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
>> >
>> >
>> > So maybe it is time to make it schedulable, e.g. one per two years or so.
>> > This one could help us to avoid such a discussion in the future, like "We
>> > don't know when Airflow 4 is coming.". At the moment when the new major
>> > version will be released new features wouldn't be added in the old major
>> > version, however we would support bug / security for a while, e.g. 1 year
>> > for bug fixes, 3 years for security fixes with a total 5 year lifecycle
>> per
>> > a major version. These just are approximate time periods for a definition
>> > of current period, bugfix period and security fix period.
>> >
>> > In contributors' perspective it helps with dropping the deprecated stuff
>> > which resolves some old problem: we have to support everything including
>> > deprecated stuff and without schedulable lifecycle for the deprecated
>> stuff
>> > it could be showstopper for the new feature, because sometimes it hard to
>> > support two different approaches for long period of time with no hope
>> that
>> > it will happen soon. For some fundamental stuff which do not require a
>> lot
>> > things time to support we could postponed removal for next after the next
>> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
>> >
>> > In the user perspective, they have at least bug fix support for a while,
>> if
>> > someone want to use legacy version it their choice, however no new
>> > features, no new version of providers (after one year)
>> >
>> >
>> > ----
>> > Best Wishes
>> > *Andrey Anshin*
>> >
>> >
>> >
>> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bd...@gmail.com> wrote:
>> >
>> > > I have left several comments :-). And on interactive dag runs even
>> after
>> > > the explanation of Vikram I still don't have a clue what we want to
>> > > accomplish there :-P.
>> > >
>> > > I would like to see a mantra or team for Airflow 3. That helps nudging
>> > > people in the same direction. Suggestions in the comments.
>> > >
>> > > Bolke
>> > > Sent from my iPhone
>> > >
>> > > > On 4 May 2024, at 01:14, Vikram Koka <vi...@astronomer.io.invalid>
>> > > wrote:
>> > > >
>> > > > Good point Jed.
>> > > > I responded back to your comment in the doc as well and very open to
>> > > > changing the term in the doc.
>> > > >
>> > > > Used the term "interactive DAG run" as the ability to invoke or
>> > trigger a
>> > > > DAG run through the API, with the expectation of getting back a
>> result
>> > > > immediately. An alternate term could be a "synchronous DAG run".
>> > > >
>> > > > Regardless, this is a significant change so a good term to indicate
>> the
>> > > > expansion from "batch runs only" is warranted. Very open to different
>> > > terms
>> > > > here.
>> > > >
>> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
>> > jedcunningham@apache.org
>> > > >
>> > > >> wrote:
>> > > >>
>> > > >> Very exciting! Looks like we will have a busy period of time ahead
>> of
>> > > us.
>> > > >> Overall I like the plan so far, especially using this year's Airflow
>> > > Summit
>> > > >> as an opportunity to announce and gather feedback, and the 2025
>> > version
>> > > to
>> > > >> pitch upgrading.
>> > > >>
>> > > >> I left a comment in the doc, but we might want to iterate on the
>> > > >> terminology we use for high priority or "synchronous" DAG runs to
>> > serve
>> > > LLM
>> > > >> responses - I find "interactive DAG runs" a bit confusing.
>> > > >>
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>> > >
>> > >
>> >
>>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Michał Modras <mi...@google.com.INVALID>]

+1 to Jens's & Bolke's points here and in the doc

I agree we should work on clarifying the directions we would like Airflow
to go. Introducing a new major Airflow version is a massive overhead for
users, who would need to plan for migrations, onboarding the new Airflow
(with a slightly different architecture), etc., and effectively Airflow 2
would live in parallel for a long time.

Personally, I think most of the points in Kaxil's/Vikram's doc are valuable
projects of their own, and I could imagine all of them being delivered as
separate AIPs within Airflow 2 (surely new minor versions of Airflow 2). I
am not sure if the scope of changes and the goal we want to achieve is a)
clear enough b) broad enough to call for a new major version.

Best,
Michal

On Sun, May 5, 2024 at 10:10 AM Scheffler Jens (XC-AS/EAE-ADA-T)
<Je...@de.bosch.com.invalid> wrote:

> Thanks for the document write-up, Kaxil. I assume this is mostly a vision
> statement.
>
> Looking forward for a larger addendum where we can collect things that we
> all can vote and agree on as targets.
>
> As I started earlier with a confluence page and it seems this is not
> accessible to all, shall we convert this to a Google Doc for better
> collaboration and item collection?
>
> Sent from Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Vikram Koka <vi...@astronomer.io.INVALID>
> Sent: Sunday, May 5, 2024 3:34:33 AM
> To: dev@airflow.apache.org <de...@airflow.apache.org>
> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
> strategic (Airflow 3) approach
>
> Thank you for your feedback, Bolke and Andrey!
>
> Bolke,
> I have replied to some of your comments in the doc.
> I will provide a detailed write up on the "Interactive DAG run" (or
> synchronous DAG run) capability, which has generated some early questions.
> I had intended to get an AIP published for that as a follow-up, but I
> believe that a simpler write up would be useful ahead of the AIP.
>
> Andrey,
> You raise an interesting point.
>
> As part of the Airflow 2.0 release, we as a community had decided to
> strictly adhere to Semver as detailed in the document you referenced. We
> also consciously split out the "Core Airflow" releases from the "Provider"
> releases at that time. We had a clear expectation then for the cadence of
> both minor and patch releases, which we have generally adhered to since
> then.
>
> Personally, I am more concerned about our Provider releases right now, as
> compared to the cadence of our major releases. I believe that one of the
> proposed changes in the Airflow 3 document i.e. the clear separation for
> Task Execution will help here, but more may be needed.
>
> Definitely interested in more feedback on this as well.
>
> Vikram
>
>
> On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <an...@taragol.is>
> wrote:
>
> > I would like to propose to change (at least discuss) release policy
> around
> > the Major version of Airflow.
> >
> > Right now it is described as "These releases do not happen with any
> regular
> > interval or on any predictable schedule." :
> >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0
> <
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
> >
> >
> > So maybe it is time to make it schedulable, e.g. one per two years or so.
> > This one could help us to avoid such a discussion in the future, like "We
> > don't know when Airflow 4 is coming.". At the moment when the new major
> > version will be released new features wouldn't be added in the old major
> > version, however we would support bug / security for a while, e.g. 1 year
> > for bug fixes, 3 years for security fixes with a total 5 year lifecycle
> per
> > a major version. These just are approximate time periods for a definition
> > of current period, bugfix period and security fix period.
> >
> > In contributors' perspective it helps with dropping the deprecated stuff
> > which resolves some old problem: we have to support everything including
> > deprecated stuff and without schedulable lifecycle for the deprecated
> stuff
> > it could be showstopper for the new feature, because sometimes it hard to
> > support two different approaches for long period of time with no hope
> that
> > it will happen soon. For some fundamental stuff which do not require a
> lot
> > things time to support we could postponed removal for next after the next
> > release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
> >
> > In the user perspective, they have at least bug fix support for a while,
> if
> > someone want to use legacy version it their choice, however no new
> > features, no new version of providers (after one year)
> >
> >
> > ----
> > Best Wishes
> > *Andrey Anshin*
> >
> >
> >
> > On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bd...@gmail.com> wrote:
> >
> > > I have left several comments :-). And on interactive dag runs even
> after
> > > the explanation of Vikram I still don't have a clue what we want to
> > > accomplish there :-P.
> > >
> > > I would like to see a mantra or team for Airflow 3. That helps nudging
> > > people in the same direction. Suggestions in the comments.
> > >
> > > Bolke
> > > Sent from my iPhone
> > >
> > > > On 4 May 2024, at 01:14, Vikram Koka <vi...@astronomer.io.invalid>
> > > wrote:
> > > >
> > > > Good point Jed.
> > > > I responded back to your comment in the doc as well and very open to
> > > > changing the term in the doc.
> > > >
> > > > Used the term "interactive DAG run" as the ability to invoke or
> > trigger a
> > > > DAG run through the API, with the expectation of getting back a
> result
> > > > immediately. An alternate term could be a "synchronous DAG run".
> > > >
> > > > Regardless, this is a significant change so a good term to indicate
> the
> > > > expansion from "batch runs only" is warranted. Very open to different
> > > terms
> > > > here.
> > > >
> > > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> > jedcunningham@apache.org
> > > >
> > > >> wrote:
> > > >>
> > > >> Very exciting! Looks like we will have a busy period of time ahead
> of
> > > us.
> > > >> Overall I like the plan so far, especially using this year's Airflow
> > > Summit
> > > >> as an opportunity to announce and gather feedback, and the 2025
> > version
> > > to
> > > >> pitch upgrading.
> > > >>
> > > >> I left a comment in the doc, but we might want to iterate on the
> > > >> terminology we use for high priority or "synchronous" DAG runs to
> > serve
> > > LLM
> > > >> responses - I find "interactive DAG runs" a bit confusing.
> > > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

["Scheffler Jens (XC-AS/EAE-ADA-T)" <Je...@de.bosch.com.INVALID>]

Thanks for the document write-up, Kaxil. I assume this is mostly a vision statement.

Looking forward for a larger addendum where we can collect things that we all can vote and agree on as targets.

As I started earlier with a confluence page and it seems this is not accessible to all, shall we convert this to a Google Doc for better collaboration and item collection?

Sent from Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Vikram Koka <vi...@astronomer.io.INVALID>
Sent: Sunday, May 5, 2024 3:34:33 AM
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

Thank you for your feedback, Bolke and Andrey!

Bolke,
I have replied to some of your comments in the doc.
I will provide a detailed write up on the "Interactive DAG run" (or
synchronous DAG run) capability, which has generated some early questions.
I had intended to get an AIP published for that as a follow-up, but I
believe that a simpler write up would be useful ahead of the AIP.

Andrey,
You raise an interesting point.

As part of the Airflow 2.0 release, we as a community had decided to
strictly adhere to Semver as detailed in the document you referenced. We
also consciously split out the "Core Airflow" releases from the "Provider"
releases at that time. We had a clear expectation then for the cadence of
both minor and patch releases, which we have generally adhered to since
then.

Personally, I am more concerned about our Provider releases right now, as
compared to the cadence of our major releases. I believe that one of the
proposed changes in the Airflow 3 document i.e. the clear separation for
Task Execution will help here, but more may be needed.

Definitely interested in more feedback on this as well.

Vikram


On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <an...@taragol.is>
wrote:

> I would like to propose to change (at least discuss) release policy around
> the Major version of Airflow.
>
> Right now it is described as "These releases do not happen with any regular
> interval or on any predictable schedule." :
>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fairflow.apache.org%2Fdocs%2Fapache-airflow%2Fstable%2Frelease-process.html%23term-Major-release&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C789cc98bb82b41e6080208dc6ca3a6ef%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638504697343083297%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=1OdyNadtakyhq4%2FQiDu1ooNaP7YOfuc7UtpU6sltPLQ%3D&reserved=0<https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release>
>
> So maybe it is time to make it schedulable, e.g. one per two years or so.
> This one could help us to avoid such a discussion in the future, like "We
> don't know when Airflow 4 is coming.". At the moment when the new major
> version will be released new features wouldn't be added in the old major
> version, however we would support bug / security for a while, e.g. 1 year
> for bug fixes, 3 years for security fixes with a total 5 year lifecycle per
> a major version. These just are approximate time periods for a definition
> of current period, bugfix period and security fix period.
>
> In contributors' perspective it helps with dropping the deprecated stuff
> which resolves some old problem: we have to support everything including
> deprecated stuff and without schedulable lifecycle for the deprecated stuff
> it could be showstopper for the new feature, because sometimes it hard to
> support two different approaches for long period of time with no hope that
> it will happen soon. For some fundamental stuff which do not require a lot
> things time to support we could postponed removal for next after the next
> release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
>
> In the user perspective, they have at least bug fix support for a while, if
> someone want to use legacy version it their choice, however no new
> features, no new version of providers (after one year)
>
>
> ----
> Best Wishes
> *Andrey Anshin*
>
>
>
> On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bd...@gmail.com> wrote:
>
> > I have left several comments :-). And on interactive dag runs even after
> > the explanation of Vikram I still don't have a clue what we want to
> > accomplish there :-P.
> >
> > I would like to see a mantra or team for Airflow 3. That helps nudging
> > people in the same direction. Suggestions in the comments.
> >
> > Bolke
> > Sent from my iPhone
> >
> > > On 4 May 2024, at 01:14, Vikram Koka <vi...@astronomer.io.invalid>
> > wrote:
> > >
> > > Good point Jed.
> > > I responded back to your comment in the doc as well and very open to
> > > changing the term in the doc.
> > >
> > > Used the term "interactive DAG run" as the ability to invoke or
> trigger a
> > > DAG run through the API, with the expectation of getting back a result
> > > immediately. An alternate term could be a "synchronous DAG run".
> > >
> > > Regardless, this is a significant change so a good term to indicate the
> > > expansion from "batch runs only" is warranted. Very open to different
> > terms
> > > here.
> > >
> > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> jedcunningham@apache.org
> > >
> > >> wrote:
> > >>
> > >> Very exciting! Looks like we will have a busy period of time ahead of
> > us.
> > >> Overall I like the plan so far, especially using this year's Airflow
> > Summit
> > >> as an opportunity to announce and gather feedback, and the 2025
> version
> > to
> > >> pitch upgrading.
> > >>
> > >> I left a comment in the doc, but we might want to iterate on the
> > >> terminology we use for high priority or "synchronous" DAG runs to
> serve
> > LLM
> > >> responses - I find "interactive DAG runs" a bit confusing.
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Vikram Koka <vi...@astronomer.io.INVALID>]

Thank you for your feedback, Bolke and Andrey!

Bolke,
I have replied to some of your comments in the doc.
I will provide a detailed write up on the "Interactive DAG run" (or
synchronous DAG run) capability, which has generated some early questions.
I had intended to get an AIP published for that as a follow-up, but I
believe that a simpler write up would be useful ahead of the AIP.

Andrey,
You raise an interesting point.

As part of the Airflow 2.0 release, we as a community had decided to
strictly adhere to Semver as detailed in the document you referenced. We
also consciously split out the "Core Airflow" releases from the "Provider"
releases at that time. We had a clear expectation then for the cadence of
both minor and patch releases, which we have generally adhered to since
then.

Personally, I am more concerned about our Provider releases right now, as
compared to the cadence of our major releases. I believe that one of the
proposed changes in the Airflow 3 document i.e. the clear separation for
Task Execution will help here, but more may be needed.

Definitely interested in more feedback on this as well.

Vikram


On Sat, May 4, 2024 at 10:57 AM Andrey Anshin <an...@taragol.is>
wrote:

> I would like to propose to change (at least discuss) release policy around
> the Major version of Airflow.
>
> Right now it is described as "These releases do not happen with any regular
> interval or on any predictable schedule." :
>
> https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release
>
> So maybe it is time to make it schedulable, e.g. one per two years or so.
> This one could help us to avoid such a discussion in the future, like "We
> don't know when Airflow 4 is coming.". At the moment when the new major
> version will be released new features wouldn't be added in the old major
> version, however we would support bug / security for a while, e.g. 1 year
> for bug fixes, 3 years for security fixes with a total 5 year lifecycle per
> a major version. These just are approximate time periods for a definition
> of current period, bugfix period and security fix period.
>
> In contributors' perspective it helps with dropping the deprecated stuff
> which resolves some old problem: we have to support everything including
> deprecated stuff and without schedulable lifecycle for the deprecated stuff
> it could be showstopper for the new feature, because sometimes it hard to
> support two different approaches for long period of time with no hope that
> it will happen soon. For some fundamental stuff which do not require a lot
> things time to support we could postponed removal for next after the next
> release, e.g. deprecate in Airflow 3, but remove it in Airflow 5
>
> In the user perspective, they have at least bug fix support for a while, if
> someone want to use legacy version it their choice, however no new
> features, no new version of providers (after one year)
>
>
> ----
> Best Wishes
> *Andrey Anshin*
>
>
>
> On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bd...@gmail.com> wrote:
>
> > I have left several comments :-). And on interactive dag runs even after
> > the explanation of Vikram I still don't have a clue what we want to
> > accomplish there :-P.
> >
> > I would like to see a mantra or team for Airflow 3. That helps nudging
> > people in the same direction. Suggestions in the comments.
> >
> > Bolke
> > Sent from my iPhone
> >
> > > On 4 May 2024, at 01:14, Vikram Koka <vi...@astronomer.io.invalid>
> > wrote:
> > >
> > > Good point Jed.
> > > I responded back to your comment in the doc as well and very open to
> > > changing the term in the doc.
> > >
> > > Used the term "interactive DAG run" as the ability to invoke or
> trigger a
> > > DAG run through the API, with the expectation of getting back a result
> > > immediately. An alternate term could be a "synchronous DAG run".
> > >
> > > Regardless, this is a significant change so a good term to indicate the
> > > expansion from "batch runs only" is warranted. Very open to different
> > terms
> > > here.
> > >
> > >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <
> jedcunningham@apache.org
> > >
> > >> wrote:
> > >>
> > >> Very exciting! Looks like we will have a busy period of time ahead of
> > us.
> > >> Overall I like the plan so far, especially using this year's Airflow
> > Summit
> > >> as an opportunity to announce and gather feedback, and the 2025
> version
> > to
> > >> pitch upgrading.
> > >>
> > >> I left a comment in the doc, but we might want to iterate on the
> > >> terminology we use for high priority or "synchronous" DAG runs to
> serve
> > LLM
> > >> responses - I find "interactive DAG runs" a bit confusing.
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Andrey Anshin <an...@taragol.is>]

I would like to propose to change (at least discuss) release policy around
the Major version of Airflow.

Right now it is described as "These releases do not happen with any regular
interval or on any predictable schedule." :
https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#term-Major-release

So maybe it is time to make it schedulable, e.g. one per two years or so.
This one could help us to avoid such a discussion in the future, like "We
don't know when Airflow 4 is coming.". At the moment when the new major
version will be released new features wouldn't be added in the old major
version, however we would support bug / security for a while, e.g. 1 year
for bug fixes, 3 years for security fixes with a total 5 year lifecycle per
a major version. These just are approximate time periods for a definition
of current period, bugfix period and security fix period.

In contributors' perspective it helps with dropping the deprecated stuff
which resolves some old problem: we have to support everything including
deprecated stuff and without schedulable lifecycle for the deprecated stuff
it could be showstopper for the new feature, because sometimes it hard to
support two different approaches for long period of time with no hope that
it will happen soon. For some fundamental stuff which do not require a lot
things time to support we could postponed removal for next after the next
release, e.g. deprecate in Airflow 3, but remove it in Airflow 5

In the user perspective, they have at least bug fix support for a while, if
someone want to use legacy version it their choice, however no new
features, no new version of providers (after one year)


----
Best Wishes
*Andrey Anshin*



On Sat, 4 May 2024 at 19:17, Bolke de Bruin <bd...@gmail.com> wrote:

> I have left several comments :-). And on interactive dag runs even after
> the explanation of Vikram I still don't have a clue what we want to
> accomplish there :-P.
>
> I would like to see a mantra or team for Airflow 3. That helps nudging
> people in the same direction. Suggestions in the comments.
>
> Bolke
> Sent from my iPhone
>
> > On 4 May 2024, at 01:14, Vikram Koka <vi...@astronomer.io.invalid>
> wrote:
> >
> > Good point Jed.
> > I responded back to your comment in the doc as well and very open to
> > changing the term in the doc.
> >
> > Used the term "interactive DAG run" as the ability to invoke or trigger a
> > DAG run through the API, with the expectation of getting back a result
> > immediately. An alternate term could be a "synchronous DAG run".
> >
> > Regardless, this is a significant change so a good term to indicate the
> > expansion from "batch runs only" is warranted. Very open to different
> terms
> > here.
> >
> >> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <jedcunningham@apache.org
> >
> >> wrote:
> >>
> >> Very exciting! Looks like we will have a busy period of time ahead of
> us.
> >> Overall I like the plan so far, especially using this year's Airflow
> Summit
> >> as an opportunity to announce and gather feedback, and the 2025 version
> to
> >> pitch upgrading.
> >>
> >> I left a comment in the doc, but we might want to iterate on the
> >> terminology we use for high priority or "synchronous" DAG runs to serve
> LLM
> >> responses - I find "interactive DAG runs" a bit confusing.
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Bolke de Bruin <bd...@gmail.com>]

I have left several comments :-). And on interactive dag runs even after the explanation of Vikram I still don't have a clue what we want to accomplish there :-P.

I would like to see a mantra or team for Airflow 3. That helps nudging people in the same direction. Suggestions in the comments.

Bolke
Sent from my iPhone

> On 4 May 2024, at 01:14, Vikram Koka <vi...@astronomer.io.invalid> wrote:
> 
> Good point Jed.
> I responded back to your comment in the doc as well and very open to
> changing the term in the doc.
> 
> Used the term "interactive DAG run" as the ability to invoke or trigger a
> DAG run through the API, with the expectation of getting back a result
> immediately. An alternate term could be a "synchronous DAG run".
> 
> Regardless, this is a significant change so a good term to indicate the
> expansion from "batch runs only" is warranted. Very open to different terms
> here.
> 
>> On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <je...@apache.org>
>> wrote:
>> 
>> Very exciting! Looks like we will have a busy period of time ahead of us.
>> Overall I like the plan so far, especially using this year's Airflow Summit
>> as an opportunity to announce and gather feedback, and the 2025 version to
>> pitch upgrading.
>> 
>> I left a comment in the doc, but we might want to iterate on the
>> terminology we use for high priority or "synchronous" DAG runs to serve LLM
>> responses - I find "interactive DAG runs" a bit confusing.
>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Vikram Koka <vi...@astronomer.io.INVALID>]

Good point Jed.
I responded back to your comment in the doc as well and very open to
changing the term in the doc.

Used the term "interactive DAG run" as the ability to invoke or trigger a
DAG run through the API, with the expectation of getting back a result
immediately. An alternate term could be a "synchronous DAG run".

Regardless, this is a significant change so a good term to indicate the
expansion from "batch runs only" is warranted. Very open to different terms
here.

On Fri, May 3, 2024 at 4:05 PM Jed Cunningham <je...@apache.org>
wrote:

> Very exciting! Looks like we will have a busy period of time ahead of us.
> Overall I like the plan so far, especially using this year's Airflow Summit
> as an opportunity to announce and gather feedback, and the 2025 version to
> pitch upgrading.
>
> I left a comment in the doc, but we might want to iterate on the
> terminology we use for high priority or "synchronous" DAG runs to serve LLM
> responses - I find "interactive DAG runs" a bit confusing.
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Jed Cunningham <je...@apache.org>]

Very exciting! Looks like we will have a busy period of time ahead of us.
Overall I like the plan so far, especially using this year's Airflow Summit
as an opportunity to announce and gather feedback, and the 2025 version to
pitch upgrading.

I left a comment in the doc, but we might want to iterate on the
terminology we use for high priority or "synchronous" DAG runs to serve LLM
responses - I find "interactive DAG runs" a bit confusing.

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Kaxil Naik <ka...@gmail.com>]

Hi all,

As promised, we are pleased to share our proposal for Airflow 3
<https://docs.google.com/document/d/1MTr53101EISZaYidCUKcR6mRKshXGzW6DZFXGzetG3E/edit?usp=sharing>.


We met several community members in the last few days to get feedback on
this proposal, and we are glad to say that most of the things in the doc
resonated with them. It was also very pleasing to see that everyone we
talked to asked how they could help contribute towards Airflow 3 and making
it successful.

We would like to now open it up for feedback from the entire community.
Please add comments to this Google doc. This proposal is purposefully
high-level to get feedback on the general direction and we will have
several AIPs for the big pieces mentioned in the doc.

If there aren't any strong objections on why we (as the Airflow community)
should work on Airflow 3, we propose a dedicated fortnightly recurring
call starting the first week of June. This will give enough time to get
feedback on our proposal, incorporate any feedback and then focus our
discussion on the What & How of Airflow 3 as opposed to Why.

As we hit the 10-year mark at this year’s Airflow Summit, we have a unique
marketing opportunity to officially announce Airflow 3. We can either use
this milestone to just look back at a decade of growth or make it more
exciting to not only showcase growth but innovation and, more importantly,
get the Airflow users excited about the next chapter in Airflow’s story.
We’ll have the perfect platform to gather immediate feedback, engage the
community in shaping future features, and share our vision for what lies
ahead. We can then focus Airflow Summit 2025 to discuss how Airflow 3
features are used in the wild by companies around the world, showcase the
migration tools and utilities that will have matured by then, and gather
more community insights to continue improving Airflow. This will give our
users more confidence in Airflow 3 and help them feel comfortable upgrading
to it. This is, of course, just our proposal, and we would love to hear
what others think. It’s certainly ambitious, but nothing we haven’t done in
the past, and having a goal-post will help us all.

Hope you all have a great weekend.

Regards,
Kaxil, Vikram & Constance

On Mon, 29 Apr 2024 at 05:11, Amogh Desai <am...@gmail.com> wrote:

> Thanks for starting the discussion, Jarek!
>
> I too agree that with the new upcoming features and AIPs, it might just be
> the right time
> to discuss the possibility of having Airflow 3. I agree with most reasons
> pointed out by others and
> I would love to see it happen, and also be a part of it.
>
> Since this is a major step for the future of Airflow, we need to carefully
> consider the user experience for
> users coming from versions of Airflow 2 and would not want this migration
> to be a pain.
>
> Btw, I concur with Jens and I too am not very clear when we say that "Gen
> AI is going to be the new trigger for Airflow".
> Would be obliged if someone could explain that portion to me :)
>
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Mon, Apr 22, 2024 at 1:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Just one comment here - while maybe "shocking" for some cases - yes, this
> > one has been clearly coming. Actually, it took a lot of my brain cycles
> > recently to think "what's next". Too much, to the point that I started
> the
> > thread.
> > I thought it might be quite a valuable opening from someone who always
> said
> > "well, we have to have **really** good reason to do Airflow 3" and "maybe
> > there will not be Airflow 3".
> >
> > And I quite agree with Kaxil - that trying to organise our thoughts
> around
> > what to do and how our Approach for Airflow 3 based on just this thread
> is
> > a bit too early.
> > I do not think this one thread here will lead to us deciding what to do -
> > if we try to do it now in a discussion thread or even a confluence doc,
> we
> > might fail achieving the goal.
> >
> > My main point here was to really get the feel and open thoughts of those
> > who are actively involved in Airflow - on what we should do next. And to
> > see if this is the right time to start thinking in "two" modes: Airflow 2
> > and Airflow 3 (even if we do not know yet what Airflow 3 will be).
> >
> > I'd rather let a free stream of thoughts of what people think should
> happen
> > here continue. Merely opening our minds to the possibility of Airflow 3.
> > And I would love to keep it flowing for others - without the goal of
> > organizing it or achieving consensus.
> >
> > And I think all that Kaxil writes about - starting a series of calls,
> > organizing our discussions, getting "product manager(s)" working on
> > organizing those discussions is the **right** thing to do.
> > How exactly to do that, how to make sure everyone is involved, while we
> are
> > not tied up in endless discussions and bike-shedding, should materialize
> > from our discussion.
> >
> > But I would propose (and encourage) others' thoughts here as well - just
> a
> > free stream of those - then it might provide valuable feedback to what's
> > next.
> >
> > J.
> >
> >
> > On Mon, Apr 22, 2024 at 4:14 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Hello all,
> > >
> > > I didn't anticipate reading an Airflow 3 email from a sunny beach in
> > Nice,
> > > France <https://en.wikipedia.org/wiki/Nice> -- I had a great time
> there
> > > over the weekend, highly recommended :D
> > >
> > > I say that because, as Vikram pointed out, some of us at Astronomer
> have
> > > been polishing up the doc to propose Airflow 3 to the community in the
> > > coming week. Such is the beauty of the open-source project that
> multiple
> > > people (in the form of developers, committers, PMC members and various
> > > Stakeholders) think the same. From the Astronomer front, Constance had
> > been
> > > championing a doc with Vikram & myself, with inputs from various other
> > > committers & users, to have a good blend of different perspectives --
> > > Product, PMC member & Industry leaders that cover several areas from
> > > User-facing pain points, Industry trends in the Orchestration space,
> > > Innovation in AI & ML space & opportunities as well as maintainability
> of
> > > the current codebase. We would love to share it this week that goes
> into
> > > some of the details to share our perspective on Why Airflow 3.0 & Why
> now
> > > etc.
> > >
> > > I would like to reiterate my statement from last year's panel session
> at
> > > the Airflow Summit with Marc, Jarek & Pierre
> > > <https://airflowsummit.org/sessions/2023/panels/panel-faces-airflow/>:
> > > "We,
> > > as the Airflow project, have maintained a great balance of Innovation &
> > > Stability", and I truly believe in that and is clearly visible in the
> > > number of downloads and Airflow's popularity as the leader in the
> > Workflow
> > > Orchestration space. Our industry is rapidly evolving, especially in
> the
> > > Data, AI & ML space. The role and expectations of the Data Orchestrator
> > > (more specialized than a generic Workflow Orchestrator) are also
> evolving
> > > as it is more and more utilized for Business critical applications than
> > > just powering dashboards. So, IMO, we must continue catering to those
> > > use-cases and innovate to aid the new use cases. Some of us from
> > Astronomer
> > > would like to create AIPs around these new use-cases LLM/Gen-AI, Data
> > > Awareness, and some of the other things discussed above in the coming
> > weeks
> > > to receive feedback from everyone.
> > >
> > > Apart from the new use cases around Data, AI & ML, balancing them with
> > > resolving user pain points with things like DAG Versioning, a more
> modern
> > > and extensible UI, lack of permissions with Airflow CLI, simplifying
> the
> > > first-user Learning Curve etc while cleaning up Tech-debt (example we
> > have
> > > around 100+ deprecations in our code-base), a more performant scheduler
> > > like the async SqlAlchemy & Scheduler discussions on the mailing list,
> > > dropping dependency on FAB, rethinking provider/core separation -- both
> > for
> > > users & developers -- will make for a powerful 3.0 release that users
> > will
> > > want to update and will provide a cleaner code-base for the
> contributors
> > to
> > > build new foundational pieces.
> > >
> > > Needless to say, similar to Airflow 2.0, we will have to provide our
> > users
> > > with utilities like the Airflow Upgrade check
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/howto/upgrading-from-1-10/upgrade-check.html
> > > >
> > > script
> > > & other tools and docs to ease the migration.
> > >
> > > Regarding the proposal to move the discussion to Confluence: In my
> > opinion,
> > > Confluence is a good place once things become more concrete and
> defined,
> > > and we are looking for feedback. For a thing as big as Airflow 3, I
> would
> > > humbly suggest the same route as what we did for Airflow 2 -- to have
> > > a few recurring
> > > Dev calls
> > > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes> to
> > > gather areas of interest and various upcoming AIPs from different
> > > stakeholders and get aligned on Why Airflow 3? and Defining the
> > high-level
> > > scope
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158869992
> > > >.
> > > This way, we can iterate much faster, and since all of the calls will
> be
> > > recorded and summary notes will be added on Confluence (example
> > > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes>)
> and
> > > posted on the mailing list
> > > <
> https://lists.apache.org/list?dev@airflow.apache.org:2020-9:dev%20call
> > >,
> > > we will have a written record of the important things. For any decision
> > > points, we will bring things to the mailing list for Vote or Lazy
> > > consensus. We could also re-use the Town Hall calls if needed. Once we
> > have
> > > good enough alignment and scope,  utilizing Confluence and mailing list
> > on
> > > the individual items & AIPs would be more valuable, in my opinion.
> > >
> > > PS: It makes me very excited that we are discussing a new major phase
> of
> > > Airflow. Looking forward to it.
> > >
> > > Regards,
> > > Kaxil
> > >
> > > On Sun, 21 Apr 2024 at 17:21, Scheffler Jens (XC-AS/EAE-ADA-T)
> > > <Je...@de.bosch.com.invalid> wrote:
> > >
> > > > Hi Developers,
> > > >
> > > > TLDR Summary: I propose to move the discussion from a Email
> > Replay-to-all
> > > > chain to a discussion collection in
> > > > https://cwiki.apache.org/confluence/x/hQv9EQ
> > > >
> > > > When I first saw this email from Jarek I was a bit surprised and
> > actually
> > > > the email was pulling me out of a kind of comfort zone, knowing what
> > are
> > > > the next steps. Naturally I was shocked a bit. So I decided to have a
> > > sleep
> > > > over it. (Might be a bit shocking because Jarek dropped it 😃
> > especially
> > > I
> > > > heard his position about a 3.0 before and that always sounded to me
> > like
> > > a
> > > > strong position... haha)
> > > > After having a sleep-over the post I think it is valid to raise the
> > > > discussion. Especially as we are going to a 10th feature-release
> which
> > > was
> > > > also a cut-over from 1.x to 2.x. At some point every software product
> > > needs
> > > > a re-factoring and cleanup. Structures are never perfect. But a lot
> of
> > > > emotions and work are included with such a step. And a risk to fail
> and
> > > to
> > > > lose a lot of users and force them to migrate (or have them
> run-away).
> > So
> > > > my current outcome is: We should carefully consider. But we need to
> > > > consider.
> > > >
> > > > I believe the discussion will take a moment and focus - and a
> > > Reply-to-all
> > > > chain will not be a good path as we will lose a lot of detail and
> focus
> > > and
> > > > emails will create a lot of noise which is hard to follow. In a
> perfect
> > > > non-distributed world I'd call you to a half-day visioning workshop
> in
> > a
> > > > room and focus on the whiteboard. Not possible with this level of
> > > > distribution. Next option would be a (large) ~4h conference call
> which
> > is
> > > > hard to make in a time-zone matching the sleep cycle for all. Perfect
> > > would
> > > > be if Summit would be close-by and plan a 1/2 day or full-day
> breakout
> > > for
> > > > contributors on Day4 or so. But September is far far away.
> > > >
> > > > Therefore - to reduce amount of emails - I propose to start points,
> > > ideas,
> > > > pain points etc. first on a Confluence page. Therefore I tried to
> start
> > > one
> > > > page as starting points (contrary ideas welcome!) to have a place to
> > > > collaborate and sketch. A virtual whiteboard would also be OK but I
> had
> > > > none at my hands to share... (like Miro, Mural etc.). If we collect
> > > ideas,
> > > > points etc. on this page we can have a rather short (2h) call with
> > > > contributors in the next time to pitch and discuss the points and
> > define
> > > > follow-up steps to a plan, vote and conclusion.
> > > >
> > > > Proposed Confluence discussion page:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Discussion+and+Planning
> > > >
> > > > As a starting point I tried to import the both emails I saw in the
> > thread
> > > > into the page as starter. As it is a call to collaborate, please
> start
> > > > editing and drop your points as well.
> > > >
> > > > Towards Jarek's mentioned trigger points:
> > > > Actually the dropped AIP-68 and AIP-69 are something that in my view
> do
> > > > NOT require Airflow to get to 3.0. I would see them either "Tactical"
> > or
> > > > "just functional enhancements". AIP-68 is "just" a bit of sugar to UI
> > and
> > > > extensions to Plugin interface in my view. AIP-69 is basically
> building
> > > > something on-top, based on the concept of Hybrid Executors. As long
> as
> > we
> > > > would assume AIP-69 does not need drastical changes, maybe only small
> > > > adjustments in the core (but concept not elaborated yet). I see this
> > > mainly
> > > > as "just another Executor" that should not need breaking changes. I
> did
> > > not
> > > > want to drop these two AIP's to start a fundamental discussion but
> > rather
> > > > to bring-in a new feature each.
> > > > The points as factors that are hard to achieve in Airflow 2.x world
> are
> > > > rather the "Multi Tenancy/Team" and "Dag Versioning" which in my eyes
> > > might
> > > > be able to move faster with a 3.0.
> > > >
> > > > P.S.: I do not get the point (yet?) Why GenAI is a trigger point that
> > > > forced structural breaking changes?
> > > >
> > > > Mit freundlichen Grüßen / Best regards
> > > >
> > > > Jens Scheffler
> > > >
> > > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> > > > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
> > > > GERMANY | www.bosch.com
> > > > Tel. +49 711 811-91508 | Mobil +49 160 90417410 |
> > > > Jens.Scheffler@de.bosch.com
> > > >
> > > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> > > > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> > > > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr.
> Markus
> > > > Forschner,
> > > > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
> > > >
> > > > -----Original Message-----
> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > > Sent: Saturday, April 20, 2024 6:23 PM
> > > > To: dev@airflow.apache.org
> > > > Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
> > > > strategic (Airflow 3) approach
> > > >
> > > > A wonderful and exciting Saturday morning discussion!
> > > > Thank you Jarek for bringing the offline conversations into the
> mailing
> > > > list.
> > > >
> > > > I completely agree on the necessity of Airflow 3.
> > > > I also agree that Gen AI is the trigger i.e. the answer to "Why now"?
> > > >
> > > > Having been thinking about this for a while from a strategic
> > perspective,
> > > > as opposed to the tactical perspective of the bi-weekly and monthly
> > > > releases, I believe that our thinking as you articulated should have
> a
> > > > clear understanding of strategic vs. tactical, but I don't believe
> our
> > > > execution needs to necessarily be either or, but can actually be
> > blended.
> > > >
> > > > With that said,  I believe that there are the following four buckets
> > that
> > > > we should use as a framework for Airflow 3.
> > > >
> > > > 1. Gen AI / LLM support
> > > > 2. Airflow User Improvements
> > > > 3. Easy adoption of Airflow by new users 4. Integration improvements
> /
> > > > Provider maintainability
> > > >
> > > > Describing them in more detail below:
> > > > 1. Gen AI / LLM support
> > > > Reiterating the fact that this needs more work, I do believe this can
> > be
> > > > incremental to Airflow. As Astronomer, we have worked on the LLM
> > > Providers
> > > > which we contributed to Airflow late last year. But clearly, there is
> > so
> > > > more to do, both from building awareness of the patterns / templates
> to
> > > > use, as well as patterns to support in Airflow to make these easier
> to
> > > use
> > > > and adopt.
> > > >
> > > > 2. Airflow User Improvements
> > > > Clearly features and improvements desired by the Community are
> > important
> > > > to continue to work on to make Airflow more approachable. The top two
> > > > features which leap to mind for me here are:
> > > > 2.1 DAG Versioning - the most requested feature in the Airflow User
> > > Survey,
> > > > 2.2 Modern UI - also comes up a lot
> > > > 2.3 Different DAG distribution processes
> > > > 2.4 Different execution mechanisms
> > > > I know there are many more which I don't currently recall.
> > > >
> > > > 3. Airflow adoption
> > > > We have discussed this many times, but we absolutely need to make the
> > > > individual first-time adoption of Airflow better.
> > > > I think the most common term I recall here is the notion of "Airflow
> > > > Standalone", but whatever the term may be, an ultra quick, simple
> > install
> > > > of Airflow and the getting started experience is something we owe our
> > > > community.
> > > >
> > > > 4. Integration / Providers
> > > > The changes we made as part of Airflow 2.0 to split the Core Airflow
> > > > releases from the Provider releases was clearly a good choice and
> made
> > a
> > > > huge impact. However, the integration maintainability balanced with
> > > growth
> > > > still seems like it could use a significant set of improvements. Elad
> > > and I
> > > > spoke about this a couple of days ago as well and I don't have a
> clear
> > > set
> > > > of next steps here, but definitely worth exploring.
> > > >
> > > > Some of us at Astronomer have been discussing this quite a bit and
> > > > planning on bringing a more polished draft to the community, but an
> > > initial
> > > > discussion on a Saturday is fun as well :). We will definitely share
> > our
> > > > Airflow 3 proposal as a document with the community within the next
> > week,
> > > > as a request for comment.
> > > >
> > > >
> > > >
> > > > On Sat, Apr 20, 2024 at 1:50 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > > > Hello here,
> > > > >
> > > > > I have been thinking a lot recently and discussing with some people
> > > > > and I am more and more convinced it's about the time we - as a
> > > > > community - should start doing changes considering "Airflow 2"
> > current
> > > > and "Airflow 3" future.
> > > > >
> > > > >
> > > > > *TL;DR: I think we should seriously start work on Airflow 3 and
> > decide
> > > > > what it means for our AIPs  - to treat some of them as more
> > "tactical"
> > > > > - things that should go into Airflow 2 and some "strategic" ones -
> > > > > being foundational for Airflow 3 - with different goals and
> > criteria.*
> > > > >
> > > > > A lot of us already think that way and a lot of us have already
> > talked
> > > > > about it for quite some time, so you should treat my mail mostly
> as a
> > > > > little trigger "let's start publicly discussing what it might mean
> > for
> > > > > us and our community and let's make it clear about the target of
> the
> > > > > initiatives we do".
> > > > >
> > > > > Some might be surprised it comes from me as I've been often saying
> > "no
> > > > > Airflow 3 without a good reason" or "possibly we will have no
> Airflow
> > > > > 3", but I think (and a number of people I spoke to have similar
> > > > > opinion) we have plenty of reasons to make some bold moves now.
> > > > >
> > > > > Over the last 4 years since Airflow 2 was out, a lot has changed
> and
> > > > > we have a number of different needs that current Airflow 2 cannot
> > > > > **really** do well
> > > > >
> > > > > - LLM/Gen-AI mainly as the important trigger
> > > > > - Cloud Native is the "way to go"
> > > > > - need to submit DAGs in other ways than dropping them to a shared
> > DAG
> > > > > folder.
> > > > > - local testing and fast iteration on developing pipelines.
> > > > > - ability to run tasks with workflow with "affinity" so that they
> can
> > > > > share inputs/outputs in shared CPU/GPU memory
> > > > > - ability to integrate seamlessly with other workflow engines -
> > making
> > > > > Airflow a "workflow of workflows
> > > > > - probably way more
> > > > > - all that while keeping a lot of the strengths of Airflow 2 - such
> > as
> > > > > continuing to have the option of using the many thousands of
> > operators
> > > > > with
> > > > > 90+ providers.
> > > > >
> > > > > All those above - we could implement better if we get rid of a
> number
> > > > > of the implicit or explicit luggage we have in Airflow 2. I think
> the
> > > > > last two proposals from Jens: AIP-68 and AIP-69 reflect very much
> > that
> > > > > - both  would have been much easier and straightforward if we got
> > > > > Airflow 3 re-designed basically at a drawing board with boldly
> > > > > dropping some Airflow 3 assumptions.
> > > > > And if we implemented core airflow 3 - taking the best part of what
> > we
> > > > > have now in Airflow 2, but generally dropping the luggage  in a new
> > > > framework.
> > > > >
> > > > > And it won't be possible without breaking some fundamental
> > assumptions
> > > > > and making Airflow 3 quite heavily incompatible with Airflow 2
> > > > >
> > > > > From "my" camp - dropping the need of having the 700+ dependencies
> > for
> > > > > Airflow + all providers in a single Python interpreter, dropinnig
> > > > > dependency on Flask/Plugins/FAB would be a huge win on its own. Not
> > > > > mentioning being able to split provider's development and
> > contribution
> > > > > from airflow core (while keeping the development of providers as
> well
> > > > > and
> > > > > contributions) - this has been highly requested.
> > > > >
> > > > > And I think we have a lot of people in our community who would be
> > able
> > > > > (and would love) to do it - I think a number of us (including
> myself)
> > > > > are a bit burned out and tired of just maintaining things in
> Airflow
> > > > > in a backwards-compatible way and would jump on the opportunity to
> > > > > rebuilding Airflow.
> > > > >
> > > > > But - we of course cannot forget about Airflow 2 users. We do not
> > want
> > > > > to "stop the world" for them. We want to keep fixing things and
> > adding
> > > > > incremental changes - and those things do not necessarily super
> > > > > "future-proof". They should help  to "keep the lights on" for a
> while
> > > > > - which means that in a number of cases it could be "band-aid".
> > AIP-44
> > > > > (internal-API), AIP-67 (multi-team) are more of those.
> > > > >
> > > > > So - what I think we might want to do as a community:
> > > > >
> > > > > * start working on Airflow 3 foundations (and decide what it means
> > for
> > > > > our users and developer community). Decide what to keep, what to
> > drop,
> > > > > what to redesign, assumptions to recreate.
> > > > >
> > > > > * explicitly split the initiatives/AIPs we have to target Airflow 2
> > > > > and Airflow 3 and treat them a bit differently in terms of
> > > > > future-proofness
> > > > >
> > > > > I would love to hear your thoughts on that (bracing for the storm
> of
> > > > > those).
> > > > >
> > > > > J.
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Amogh Desai <am...@gmail.com>]

Thanks for starting the discussion, Jarek!

I too agree that with the new upcoming features and AIPs, it might just be
the right time
to discuss the possibility of having Airflow 3. I agree with most reasons
pointed out by others and
I would love to see it happen, and also be a part of it.

Since this is a major step for the future of Airflow, we need to carefully
consider the user experience for
users coming from versions of Airflow 2 and would not want this migration
to be a pain.

Btw, I concur with Jens and I too am not very clear when we say that "Gen
AI is going to be the new trigger for Airflow".
Would be obliged if someone could explain that portion to me :)


Thanks & Regards,
Amogh Desai


On Mon, Apr 22, 2024 at 1:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Just one comment here - while maybe "shocking" for some cases - yes, this
> one has been clearly coming. Actually, it took a lot of my brain cycles
> recently to think "what's next". Too much, to the point that I started the
> thread.
> I thought it might be quite a valuable opening from someone who always said
> "well, we have to have **really** good reason to do Airflow 3" and "maybe
> there will not be Airflow 3".
>
> And I quite agree with Kaxil - that trying to organise our thoughts around
> what to do and how our Approach for Airflow 3 based on just this thread is
> a bit too early.
> I do not think this one thread here will lead to us deciding what to do -
> if we try to do it now in a discussion thread or even a confluence doc, we
> might fail achieving the goal.
>
> My main point here was to really get the feel and open thoughts of those
> who are actively involved in Airflow - on what we should do next. And to
> see if this is the right time to start thinking in "two" modes: Airflow 2
> and Airflow 3 (even if we do not know yet what Airflow 3 will be).
>
> I'd rather let a free stream of thoughts of what people think should happen
> here continue. Merely opening our minds to the possibility of Airflow 3.
> And I would love to keep it flowing for others - without the goal of
> organizing it or achieving consensus.
>
> And I think all that Kaxil writes about - starting a series of calls,
> organizing our discussions, getting "product manager(s)" working on
> organizing those discussions is the **right** thing to do.
> How exactly to do that, how to make sure everyone is involved, while we are
> not tied up in endless discussions and bike-shedding, should materialize
> from our discussion.
>
> But I would propose (and encourage) others' thoughts here as well - just a
> free stream of those - then it might provide valuable feedback to what's
> next.
>
> J.
>
>
> On Mon, Apr 22, 2024 at 4:14 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Hello all,
> >
> > I didn't anticipate reading an Airflow 3 email from a sunny beach in
> Nice,
> > France <https://en.wikipedia.org/wiki/Nice> -- I had a great time there
> > over the weekend, highly recommended :D
> >
> > I say that because, as Vikram pointed out, some of us at Astronomer have
> > been polishing up the doc to propose Airflow 3 to the community in the
> > coming week. Such is the beauty of the open-source project that multiple
> > people (in the form of developers, committers, PMC members and various
> > Stakeholders) think the same. From the Astronomer front, Constance had
> been
> > championing a doc with Vikram & myself, with inputs from various other
> > committers & users, to have a good blend of different perspectives --
> > Product, PMC member & Industry leaders that cover several areas from
> > User-facing pain points, Industry trends in the Orchestration space,
> > Innovation in AI & ML space & opportunities as well as maintainability of
> > the current codebase. We would love to share it this week that goes into
> > some of the details to share our perspective on Why Airflow 3.0 & Why now
> > etc.
> >
> > I would like to reiterate my statement from last year's panel session at
> > the Airflow Summit with Marc, Jarek & Pierre
> > <https://airflowsummit.org/sessions/2023/panels/panel-faces-airflow/>:
> > "We,
> > as the Airflow project, have maintained a great balance of Innovation &
> > Stability", and I truly believe in that and is clearly visible in the
> > number of downloads and Airflow's popularity as the leader in the
> Workflow
> > Orchestration space. Our industry is rapidly evolving, especially in the
> > Data, AI & ML space. The role and expectations of the Data Orchestrator
> > (more specialized than a generic Workflow Orchestrator) are also evolving
> > as it is more and more utilized for Business critical applications than
> > just powering dashboards. So, IMO, we must continue catering to those
> > use-cases and innovate to aid the new use cases. Some of us from
> Astronomer
> > would like to create AIPs around these new use-cases LLM/Gen-AI, Data
> > Awareness, and some of the other things discussed above in the coming
> weeks
> > to receive feedback from everyone.
> >
> > Apart from the new use cases around Data, AI & ML, balancing them with
> > resolving user pain points with things like DAG Versioning, a more modern
> > and extensible UI, lack of permissions with Airflow CLI, simplifying the
> > first-user Learning Curve etc while cleaning up Tech-debt (example we
> have
> > around 100+ deprecations in our code-base), a more performant scheduler
> > like the async SqlAlchemy & Scheduler discussions on the mailing list,
> > dropping dependency on FAB, rethinking provider/core separation -- both
> for
> > users & developers -- will make for a powerful 3.0 release that users
> will
> > want to update and will provide a cleaner code-base for the contributors
> to
> > build new foundational pieces.
> >
> > Needless to say, similar to Airflow 2.0, we will have to provide our
> users
> > with utilities like the Airflow Upgrade check
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/howto/upgrading-from-1-10/upgrade-check.html
> > >
> > script
> > & other tools and docs to ease the migration.
> >
> > Regarding the proposal to move the discussion to Confluence: In my
> opinion,
> > Confluence is a good place once things become more concrete and defined,
> > and we are looking for feedback. For a thing as big as Airflow 3, I would
> > humbly suggest the same route as what we did for Airflow 2 -- to have
> > a few recurring
> > Dev calls
> > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes> to
> > gather areas of interest and various upcoming AIPs from different
> > stakeholders and get aligned on Why Airflow 3? and Defining the
> high-level
> > scope
> > <
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158869992
> > >.
> > This way, we can iterate much faster, and since all of the calls will be
> > recorded and summary notes will be added on Confluence (example
> > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes>) and
> > posted on the mailing list
> > <https://lists.apache.org/list?dev@airflow.apache.org:2020-9:dev%20call
> >,
> > we will have a written record of the important things. For any decision
> > points, we will bring things to the mailing list for Vote or Lazy
> > consensus. We could also re-use the Town Hall calls if needed. Once we
> have
> > good enough alignment and scope,  utilizing Confluence and mailing list
> on
> > the individual items & AIPs would be more valuable, in my opinion.
> >
> > PS: It makes me very excited that we are discussing a new major phase of
> > Airflow. Looking forward to it.
> >
> > Regards,
> > Kaxil
> >
> > On Sun, 21 Apr 2024 at 17:21, Scheffler Jens (XC-AS/EAE-ADA-T)
> > <Je...@de.bosch.com.invalid> wrote:
> >
> > > Hi Developers,
> > >
> > > TLDR Summary: I propose to move the discussion from a Email
> Replay-to-all
> > > chain to a discussion collection in
> > > https://cwiki.apache.org/confluence/x/hQv9EQ
> > >
> > > When I first saw this email from Jarek I was a bit surprised and
> actually
> > > the email was pulling me out of a kind of comfort zone, knowing what
> are
> > > the next steps. Naturally I was shocked a bit. So I decided to have a
> > sleep
> > > over it. (Might be a bit shocking because Jarek dropped it 😃
> especially
> > I
> > > heard his position about a 3.0 before and that always sounded to me
> like
> > a
> > > strong position... haha)
> > > After having a sleep-over the post I think it is valid to raise the
> > > discussion. Especially as we are going to a 10th feature-release which
> > was
> > > also a cut-over from 1.x to 2.x. At some point every software product
> > needs
> > > a re-factoring and cleanup. Structures are never perfect. But a lot of
> > > emotions and work are included with such a step. And a risk to fail and
> > to
> > > lose a lot of users and force them to migrate (or have them run-away).
> So
> > > my current outcome is: We should carefully consider. But we need to
> > > consider.
> > >
> > > I believe the discussion will take a moment and focus - and a
> > Reply-to-all
> > > chain will not be a good path as we will lose a lot of detail and focus
> > and
> > > emails will create a lot of noise which is hard to follow. In a perfect
> > > non-distributed world I'd call you to a half-day visioning workshop in
> a
> > > room and focus on the whiteboard. Not possible with this level of
> > > distribution. Next option would be a (large) ~4h conference call which
> is
> > > hard to make in a time-zone matching the sleep cycle for all. Perfect
> > would
> > > be if Summit would be close-by and plan a 1/2 day or full-day breakout
> > for
> > > contributors on Day4 or so. But September is far far away.
> > >
> > > Therefore - to reduce amount of emails - I propose to start points,
> > ideas,
> > > pain points etc. first on a Confluence page. Therefore I tried to start
> > one
> > > page as starting points (contrary ideas welcome!) to have a place to
> > > collaborate and sketch. A virtual whiteboard would also be OK but I had
> > > none at my hands to share... (like Miro, Mural etc.). If we collect
> > ideas,
> > > points etc. on this page we can have a rather short (2h) call with
> > > contributors in the next time to pitch and discuss the points and
> define
> > > follow-up steps to a plan, vote and conclusion.
> > >
> > > Proposed Confluence discussion page:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Discussion+and+Planning
> > >
> > > As a starting point I tried to import the both emails I saw in the
> thread
> > > into the page as starter. As it is a call to collaborate, please start
> > > editing and drop your points as well.
> > >
> > > Towards Jarek's mentioned trigger points:
> > > Actually the dropped AIP-68 and AIP-69 are something that in my view do
> > > NOT require Airflow to get to 3.0. I would see them either "Tactical"
> or
> > > "just functional enhancements". AIP-68 is "just" a bit of sugar to UI
> and
> > > extensions to Plugin interface in my view. AIP-69 is basically building
> > > something on-top, based on the concept of Hybrid Executors. As long as
> we
> > > would assume AIP-69 does not need drastical changes, maybe only small
> > > adjustments in the core (but concept not elaborated yet). I see this
> > mainly
> > > as "just another Executor" that should not need breaking changes. I did
> > not
> > > want to drop these two AIP's to start a fundamental discussion but
> rather
> > > to bring-in a new feature each.
> > > The points as factors that are hard to achieve in Airflow 2.x world are
> > > rather the "Multi Tenancy/Team" and "Dag Versioning" which in my eyes
> > might
> > > be able to move faster with a 3.0.
> > >
> > > P.S.: I do not get the point (yet?) Why GenAI is a trigger point that
> > > forced structural breaking changes?
> > >
> > > Mit freundlichen Grüßen / Best regards
> > >
> > > Jens Scheffler
> > >
> > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> > > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
> > > GERMANY | www.bosch.com
> > > Tel. +49 711 811-91508 | Mobil +49 160 90417410 |
> > > Jens.Scheffler@de.bosch.com
> > >
> > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> > > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> > > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> > > Forschner,
> > > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
> > >
> > > -----Original Message-----
> > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > Sent: Saturday, April 20, 2024 6:23 PM
> > > To: dev@airflow.apache.org
> > > Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
> > > strategic (Airflow 3) approach
> > >
> > > A wonderful and exciting Saturday morning discussion!
> > > Thank you Jarek for bringing the offline conversations into the mailing
> > > list.
> > >
> > > I completely agree on the necessity of Airflow 3.
> > > I also agree that Gen AI is the trigger i.e. the answer to "Why now"?
> > >
> > > Having been thinking about this for a while from a strategic
> perspective,
> > > as opposed to the tactical perspective of the bi-weekly and monthly
> > > releases, I believe that our thinking as you articulated should have a
> > > clear understanding of strategic vs. tactical, but I don't believe our
> > > execution needs to necessarily be either or, but can actually be
> blended.
> > >
> > > With that said,  I believe that there are the following four buckets
> that
> > > we should use as a framework for Airflow 3.
> > >
> > > 1. Gen AI / LLM support
> > > 2. Airflow User Improvements
> > > 3. Easy adoption of Airflow by new users 4. Integration improvements /
> > > Provider maintainability
> > >
> > > Describing them in more detail below:
> > > 1. Gen AI / LLM support
> > > Reiterating the fact that this needs more work, I do believe this can
> be
> > > incremental to Airflow. As Astronomer, we have worked on the LLM
> > Providers
> > > which we contributed to Airflow late last year. But clearly, there is
> so
> > > more to do, both from building awareness of the patterns / templates to
> > > use, as well as patterns to support in Airflow to make these easier to
> > use
> > > and adopt.
> > >
> > > 2. Airflow User Improvements
> > > Clearly features and improvements desired by the Community are
> important
> > > to continue to work on to make Airflow more approachable. The top two
> > > features which leap to mind for me here are:
> > > 2.1 DAG Versioning - the most requested feature in the Airflow User
> > Survey,
> > > 2.2 Modern UI - also comes up a lot
> > > 2.3 Different DAG distribution processes
> > > 2.4 Different execution mechanisms
> > > I know there are many more which I don't currently recall.
> > >
> > > 3. Airflow adoption
> > > We have discussed this many times, but we absolutely need to make the
> > > individual first-time adoption of Airflow better.
> > > I think the most common term I recall here is the notion of "Airflow
> > > Standalone", but whatever the term may be, an ultra quick, simple
> install
> > > of Airflow and the getting started experience is something we owe our
> > > community.
> > >
> > > 4. Integration / Providers
> > > The changes we made as part of Airflow 2.0 to split the Core Airflow
> > > releases from the Provider releases was clearly a good choice and made
> a
> > > huge impact. However, the integration maintainability balanced with
> > growth
> > > still seems like it could use a significant set of improvements. Elad
> > and I
> > > spoke about this a couple of days ago as well and I don't have a clear
> > set
> > > of next steps here, but definitely worth exploring.
> > >
> > > Some of us at Astronomer have been discussing this quite a bit and
> > > planning on bringing a more polished draft to the community, but an
> > initial
> > > discussion on a Saturday is fun as well :). We will definitely share
> our
> > > Airflow 3 proposal as a document with the community within the next
> week,
> > > as a request for comment.
> > >
> > >
> > >
> > > On Sat, Apr 20, 2024 at 1:50 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Hello here,
> > > >
> > > > I have been thinking a lot recently and discussing with some people
> > > > and I am more and more convinced it's about the time we - as a
> > > > community - should start doing changes considering "Airflow 2"
> current
> > > and "Airflow 3" future.
> > > >
> > > >
> > > > *TL;DR: I think we should seriously start work on Airflow 3 and
> decide
> > > > what it means for our AIPs  - to treat some of them as more
> "tactical"
> > > > - things that should go into Airflow 2 and some "strategic" ones -
> > > > being foundational for Airflow 3 - with different goals and
> criteria.*
> > > >
> > > > A lot of us already think that way and a lot of us have already
> talked
> > > > about it for quite some time, so you should treat my mail mostly as a
> > > > little trigger "let's start publicly discussing what it might mean
> for
> > > > us and our community and let's make it clear about the target of the
> > > > initiatives we do".
> > > >
> > > > Some might be surprised it comes from me as I've been often saying
> "no
> > > > Airflow 3 without a good reason" or "possibly we will have no Airflow
> > > > 3", but I think (and a number of people I spoke to have similar
> > > > opinion) we have plenty of reasons to make some bold moves now.
> > > >
> > > > Over the last 4 years since Airflow 2 was out, a lot has changed and
> > > > we have a number of different needs that current Airflow 2 cannot
> > > > **really** do well
> > > >
> > > > - LLM/Gen-AI mainly as the important trigger
> > > > - Cloud Native is the "way to go"
> > > > - need to submit DAGs in other ways than dropping them to a shared
> DAG
> > > > folder.
> > > > - local testing and fast iteration on developing pipelines.
> > > > - ability to run tasks with workflow with "affinity" so that they can
> > > > share inputs/outputs in shared CPU/GPU memory
> > > > - ability to integrate seamlessly with other workflow engines -
> making
> > > > Airflow a "workflow of workflows
> > > > - probably way more
> > > > - all that while keeping a lot of the strengths of Airflow 2 - such
> as
> > > > continuing to have the option of using the many thousands of
> operators
> > > > with
> > > > 90+ providers.
> > > >
> > > > All those above - we could implement better if we get rid of a number
> > > > of the implicit or explicit luggage we have in Airflow 2. I think the
> > > > last two proposals from Jens: AIP-68 and AIP-69 reflect very much
> that
> > > > - both  would have been much easier and straightforward if we got
> > > > Airflow 3 re-designed basically at a drawing board with boldly
> > > > dropping some Airflow 3 assumptions.
> > > > And if we implemented core airflow 3 - taking the best part of what
> we
> > > > have now in Airflow 2, but generally dropping the luggage  in a new
> > > framework.
> > > >
> > > > And it won't be possible without breaking some fundamental
> assumptions
> > > > and making Airflow 3 quite heavily incompatible with Airflow 2
> > > >
> > > > From "my" camp - dropping the need of having the 700+ dependencies
> for
> > > > Airflow + all providers in a single Python interpreter, dropinnig
> > > > dependency on Flask/Plugins/FAB would be a huge win on its own. Not
> > > > mentioning being able to split provider's development and
> contribution
> > > > from airflow core (while keeping the development of providers as well
> > > > and
> > > > contributions) - this has been highly requested.
> > > >
> > > > And I think we have a lot of people in our community who would be
> able
> > > > (and would love) to do it - I think a number of us (including myself)
> > > > are a bit burned out and tired of just maintaining things in Airflow
> > > > in a backwards-compatible way and would jump on the opportunity to
> > > > rebuilding Airflow.
> > > >
> > > > But - we of course cannot forget about Airflow 2 users. We do not
> want
> > > > to "stop the world" for them. We want to keep fixing things and
> adding
> > > > incremental changes - and those things do not necessarily super
> > > > "future-proof". They should help  to "keep the lights on" for a while
> > > > - which means that in a number of cases it could be "band-aid".
> AIP-44
> > > > (internal-API), AIP-67 (multi-team) are more of those.
> > > >
> > > > So - what I think we might want to do as a community:
> > > >
> > > > * start working on Airflow 3 foundations (and decide what it means
> for
> > > > our users and developer community). Decide what to keep, what to
> drop,
> > > > what to redesign, assumptions to recreate.
> > > >
> > > > * explicitly split the initiatives/AIPs we have to target Airflow 2
> > > > and Airflow 3 and treat them a bit differently in terms of
> > > > future-proofness
> > > >
> > > > I would love to hear your thoughts on that (bracing for the storm of
> > > > those).
> > > >
> > > > J.
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Jarek Potiuk <ja...@potiuk.com>]

Just one comment here - while maybe "shocking" for some cases - yes, this
one has been clearly coming. Actually, it took a lot of my brain cycles
recently to think "what's next". Too much, to the point that I started the
thread.
I thought it might be quite a valuable opening from someone who always said
"well, we have to have **really** good reason to do Airflow 3" and "maybe
there will not be Airflow 3".

And I quite agree with Kaxil - that trying to organise our thoughts around
what to do and how our Approach for Airflow 3 based on just this thread is
a bit too early.
I do not think this one thread here will lead to us deciding what to do -
if we try to do it now in a discussion thread or even a confluence doc, we
might fail achieving the goal.

My main point here was to really get the feel and open thoughts of those
who are actively involved in Airflow - on what we should do next. And to
see if this is the right time to start thinking in "two" modes: Airflow 2
and Airflow 3 (even if we do not know yet what Airflow 3 will be).

I'd rather let a free stream of thoughts of what people think should happen
here continue. Merely opening our minds to the possibility of Airflow 3.
And I would love to keep it flowing for others - without the goal of
organizing it or achieving consensus.

And I think all that Kaxil writes about - starting a series of calls,
organizing our discussions, getting "product manager(s)" working on
organizing those discussions is the **right** thing to do.
How exactly to do that, how to make sure everyone is involved, while we are
not tied up in endless discussions and bike-shedding, should materialize
from our discussion.

But I would propose (and encourage) others' thoughts here as well - just a
free stream of those - then it might provide valuable feedback to what's
next.

J.


On Mon, Apr 22, 2024 at 4:14 AM Kaxil Naik <ka...@gmail.com> wrote:

> Hello all,
>
> I didn't anticipate reading an Airflow 3 email from a sunny beach in Nice,
> France <https://en.wikipedia.org/wiki/Nice> -- I had a great time there
> over the weekend, highly recommended :D
>
> I say that because, as Vikram pointed out, some of us at Astronomer have
> been polishing up the doc to propose Airflow 3 to the community in the
> coming week. Such is the beauty of the open-source project that multiple
> people (in the form of developers, committers, PMC members and various
> Stakeholders) think the same. From the Astronomer front, Constance had been
> championing a doc with Vikram & myself, with inputs from various other
> committers & users, to have a good blend of different perspectives --
> Product, PMC member & Industry leaders that cover several areas from
> User-facing pain points, Industry trends in the Orchestration space,
> Innovation in AI & ML space & opportunities as well as maintainability of
> the current codebase. We would love to share it this week that goes into
> some of the details to share our perspective on Why Airflow 3.0 & Why now
> etc.
>
> I would like to reiterate my statement from last year's panel session at
> the Airflow Summit with Marc, Jarek & Pierre
> <https://airflowsummit.org/sessions/2023/panels/panel-faces-airflow/>:
> "We,
> as the Airflow project, have maintained a great balance of Innovation &
> Stability", and I truly believe in that and is clearly visible in the
> number of downloads and Airflow's popularity as the leader in the Workflow
> Orchestration space. Our industry is rapidly evolving, especially in the
> Data, AI & ML space. The role and expectations of the Data Orchestrator
> (more specialized than a generic Workflow Orchestrator) are also evolving
> as it is more and more utilized for Business critical applications than
> just powering dashboards. So, IMO, we must continue catering to those
> use-cases and innovate to aid the new use cases. Some of us from Astronomer
> would like to create AIPs around these new use-cases LLM/Gen-AI, Data
> Awareness, and some of the other things discussed above in the coming weeks
> to receive feedback from everyone.
>
> Apart from the new use cases around Data, AI & ML, balancing them with
> resolving user pain points with things like DAG Versioning, a more modern
> and extensible UI, lack of permissions with Airflow CLI, simplifying the
> first-user Learning Curve etc while cleaning up Tech-debt (example we have
> around 100+ deprecations in our code-base), a more performant scheduler
> like the async SqlAlchemy & Scheduler discussions on the mailing list,
> dropping dependency on FAB, rethinking provider/core separation -- both for
> users & developers -- will make for a powerful 3.0 release that users will
> want to update and will provide a cleaner code-base for the contributors to
> build new foundational pieces.
>
> Needless to say, similar to Airflow 2.0, we will have to provide our users
> with utilities like the Airflow Upgrade check
> <
> https://airflow.apache.org/docs/apache-airflow/stable/howto/upgrading-from-1-10/upgrade-check.html
> >
> script
> & other tools and docs to ease the migration.
>
> Regarding the proposal to move the discussion to Confluence: In my opinion,
> Confluence is a good place once things become more concrete and defined,
> and we are looking for feedback. For a thing as big as Airflow 3, I would
> humbly suggest the same route as what we did for Airflow 2 -- to have
> a few recurring
> Dev calls
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes> to
> gather areas of interest and various upcoming AIPs from different
> stakeholders and get aligned on Why Airflow 3? and Defining the high-level
> scope
> <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158869992
> >.
> This way, we can iterate much faster, and since all of the calls will be
> recorded and summary notes will be added on Confluence (example
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes>) and
> posted on the mailing list
> <https://lists.apache.org/list?dev@airflow.apache.org:2020-9:dev%20call>,
> we will have a written record of the important things. For any decision
> points, we will bring things to the mailing list for Vote or Lazy
> consensus. We could also re-use the Town Hall calls if needed. Once we have
> good enough alignment and scope,  utilizing Confluence and mailing list on
> the individual items & AIPs would be more valuable, in my opinion.
>
> PS: It makes me very excited that we are discussing a new major phase of
> Airflow. Looking forward to it.
>
> Regards,
> Kaxil
>
> On Sun, 21 Apr 2024 at 17:21, Scheffler Jens (XC-AS/EAE-ADA-T)
> <Je...@de.bosch.com.invalid> wrote:
>
> > Hi Developers,
> >
> > TLDR Summary: I propose to move the discussion from a Email Replay-to-all
> > chain to a discussion collection in
> > https://cwiki.apache.org/confluence/x/hQv9EQ
> >
> > When I first saw this email from Jarek I was a bit surprised and actually
> > the email was pulling me out of a kind of comfort zone, knowing what are
> > the next steps. Naturally I was shocked a bit. So I decided to have a
> sleep
> > over it. (Might be a bit shocking because Jarek dropped it 😃 especially
> I
> > heard his position about a 3.0 before and that always sounded to me like
> a
> > strong position... haha)
> > After having a sleep-over the post I think it is valid to raise the
> > discussion. Especially as we are going to a 10th feature-release which
> was
> > also a cut-over from 1.x to 2.x. At some point every software product
> needs
> > a re-factoring and cleanup. Structures are never perfect. But a lot of
> > emotions and work are included with such a step. And a risk to fail and
> to
> > lose a lot of users and force them to migrate (or have them run-away). So
> > my current outcome is: We should carefully consider. But we need to
> > consider.
> >
> > I believe the discussion will take a moment and focus - and a
> Reply-to-all
> > chain will not be a good path as we will lose a lot of detail and focus
> and
> > emails will create a lot of noise which is hard to follow. In a perfect
> > non-distributed world I'd call you to a half-day visioning workshop in a
> > room and focus on the whiteboard. Not possible with this level of
> > distribution. Next option would be a (large) ~4h conference call which is
> > hard to make in a time-zone matching the sleep cycle for all. Perfect
> would
> > be if Summit would be close-by and plan a 1/2 day or full-day breakout
> for
> > contributors on Day4 or so. But September is far far away.
> >
> > Therefore - to reduce amount of emails - I propose to start points,
> ideas,
> > pain points etc. first on a Confluence page. Therefore I tried to start
> one
> > page as starting points (contrary ideas welcome!) to have a place to
> > collaborate and sketch. A virtual whiteboard would also be OK but I had
> > none at my hands to share... (like Miro, Mural etc.). If we collect
> ideas,
> > points etc. on this page we can have a rather short (2h) call with
> > contributors in the next time to pitch and discuss the points and define
> > follow-up steps to a plan, vote and conclusion.
> >
> > Proposed Confluence discussion page:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Discussion+and+Planning
> >
> > As a starting point I tried to import the both emails I saw in the thread
> > into the page as starter. As it is a call to collaborate, please start
> > editing and drop your points as well.
> >
> > Towards Jarek's mentioned trigger points:
> > Actually the dropped AIP-68 and AIP-69 are something that in my view do
> > NOT require Airflow to get to 3.0. I would see them either "Tactical" or
> > "just functional enhancements". AIP-68 is "just" a bit of sugar to UI and
> > extensions to Plugin interface in my view. AIP-69 is basically building
> > something on-top, based on the concept of Hybrid Executors. As long as we
> > would assume AIP-69 does not need drastical changes, maybe only small
> > adjustments in the core (but concept not elaborated yet). I see this
> mainly
> > as "just another Executor" that should not need breaking changes. I did
> not
> > want to drop these two AIP's to start a fundamental discussion but rather
> > to bring-in a new feature each.
> > The points as factors that are hard to achieve in Airflow 2.x world are
> > rather the "Multi Tenancy/Team" and "Dag Versioning" which in my eyes
> might
> > be able to move faster with a 3.0.
> >
> > P.S.: I do not get the point (yet?) Why GenAI is a trigger point that
> > forced structural breaking changes?
> >
> > Mit freundlichen Grüßen / Best regards
> >
> > Jens Scheffler
> >
> > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
> > GERMANY | www.bosch.com
> > Tel. +49 711 811-91508 | Mobil +49 160 90417410 |
> > Jens.Scheffler@de.bosch.com
> >
> > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> > Forschner,
> > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
> >
> > -----Original Message-----
> > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > Sent: Saturday, April 20, 2024 6:23 PM
> > To: dev@airflow.apache.org
> > Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
> > strategic (Airflow 3) approach
> >
> > A wonderful and exciting Saturday morning discussion!
> > Thank you Jarek for bringing the offline conversations into the mailing
> > list.
> >
> > I completely agree on the necessity of Airflow 3.
> > I also agree that Gen AI is the trigger i.e. the answer to "Why now"?
> >
> > Having been thinking about this for a while from a strategic perspective,
> > as opposed to the tactical perspective of the bi-weekly and monthly
> > releases, I believe that our thinking as you articulated should have a
> > clear understanding of strategic vs. tactical, but I don't believe our
> > execution needs to necessarily be either or, but can actually be blended.
> >
> > With that said,  I believe that there are the following four buckets that
> > we should use as a framework for Airflow 3.
> >
> > 1. Gen AI / LLM support
> > 2. Airflow User Improvements
> > 3. Easy adoption of Airflow by new users 4. Integration improvements /
> > Provider maintainability
> >
> > Describing them in more detail below:
> > 1. Gen AI / LLM support
> > Reiterating the fact that this needs more work, I do believe this can be
> > incremental to Airflow. As Astronomer, we have worked on the LLM
> Providers
> > which we contributed to Airflow late last year. But clearly, there is so
> > more to do, both from building awareness of the patterns / templates to
> > use, as well as patterns to support in Airflow to make these easier to
> use
> > and adopt.
> >
> > 2. Airflow User Improvements
> > Clearly features and improvements desired by the Community are important
> > to continue to work on to make Airflow more approachable. The top two
> > features which leap to mind for me here are:
> > 2.1 DAG Versioning - the most requested feature in the Airflow User
> Survey,
> > 2.2 Modern UI - also comes up a lot
> > 2.3 Different DAG distribution processes
> > 2.4 Different execution mechanisms
> > I know there are many more which I don't currently recall.
> >
> > 3. Airflow adoption
> > We have discussed this many times, but we absolutely need to make the
> > individual first-time adoption of Airflow better.
> > I think the most common term I recall here is the notion of "Airflow
> > Standalone", but whatever the term may be, an ultra quick, simple install
> > of Airflow and the getting started experience is something we owe our
> > community.
> >
> > 4. Integration / Providers
> > The changes we made as part of Airflow 2.0 to split the Core Airflow
> > releases from the Provider releases was clearly a good choice and made a
> > huge impact. However, the integration maintainability balanced with
> growth
> > still seems like it could use a significant set of improvements. Elad
> and I
> > spoke about this a couple of days ago as well and I don't have a clear
> set
> > of next steps here, but definitely worth exploring.
> >
> > Some of us at Astronomer have been discussing this quite a bit and
> > planning on bringing a more polished draft to the community, but an
> initial
> > discussion on a Saturday is fun as well :). We will definitely share our
> > Airflow 3 proposal as a document with the community within the next week,
> > as a request for comment.
> >
> >
> >
> > On Sat, Apr 20, 2024 at 1:50 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Hello here,
> > >
> > > I have been thinking a lot recently and discussing with some people
> > > and I am more and more convinced it's about the time we - as a
> > > community - should start doing changes considering "Airflow 2" current
> > and "Airflow 3" future.
> > >
> > >
> > > *TL;DR: I think we should seriously start work on Airflow 3 and decide
> > > what it means for our AIPs  - to treat some of them as more "tactical"
> > > - things that should go into Airflow 2 and some "strategic" ones -
> > > being foundational for Airflow 3 - with different goals and criteria.*
> > >
> > > A lot of us already think that way and a lot of us have already talked
> > > about it for quite some time, so you should treat my mail mostly as a
> > > little trigger "let's start publicly discussing what it might mean for
> > > us and our community and let's make it clear about the target of the
> > > initiatives we do".
> > >
> > > Some might be surprised it comes from me as I've been often saying "no
> > > Airflow 3 without a good reason" or "possibly we will have no Airflow
> > > 3", but I think (and a number of people I spoke to have similar
> > > opinion) we have plenty of reasons to make some bold moves now.
> > >
> > > Over the last 4 years since Airflow 2 was out, a lot has changed and
> > > we have a number of different needs that current Airflow 2 cannot
> > > **really** do well
> > >
> > > - LLM/Gen-AI mainly as the important trigger
> > > - Cloud Native is the "way to go"
> > > - need to submit DAGs in other ways than dropping them to a shared DAG
> > > folder.
> > > - local testing and fast iteration on developing pipelines.
> > > - ability to run tasks with workflow with "affinity" so that they can
> > > share inputs/outputs in shared CPU/GPU memory
> > > - ability to integrate seamlessly with other workflow engines - making
> > > Airflow a "workflow of workflows
> > > - probably way more
> > > - all that while keeping a lot of the strengths of Airflow 2 - such as
> > > continuing to have the option of using the many thousands of operators
> > > with
> > > 90+ providers.
> > >
> > > All those above - we could implement better if we get rid of a number
> > > of the implicit or explicit luggage we have in Airflow 2. I think the
> > > last two proposals from Jens: AIP-68 and AIP-69 reflect very much that
> > > - both  would have been much easier and straightforward if we got
> > > Airflow 3 re-designed basically at a drawing board with boldly
> > > dropping some Airflow 3 assumptions.
> > > And if we implemented core airflow 3 - taking the best part of what we
> > > have now in Airflow 2, but generally dropping the luggage  in a new
> > framework.
> > >
> > > And it won't be possible without breaking some fundamental assumptions
> > > and making Airflow 3 quite heavily incompatible with Airflow 2
> > >
> > > From "my" camp - dropping the need of having the 700+ dependencies for
> > > Airflow + all providers in a single Python interpreter, dropinnig
> > > dependency on Flask/Plugins/FAB would be a huge win on its own. Not
> > > mentioning being able to split provider's development and contribution
> > > from airflow core (while keeping the development of providers as well
> > > and
> > > contributions) - this has been highly requested.
> > >
> > > And I think we have a lot of people in our community who would be able
> > > (and would love) to do it - I think a number of us (including myself)
> > > are a bit burned out and tired of just maintaining things in Airflow
> > > in a backwards-compatible way and would jump on the opportunity to
> > > rebuilding Airflow.
> > >
> > > But - we of course cannot forget about Airflow 2 users. We do not want
> > > to "stop the world" for them. We want to keep fixing things and adding
> > > incremental changes - and those things do not necessarily super
> > > "future-proof". They should help  to "keep the lights on" for a while
> > > - which means that in a number of cases it could be "band-aid". AIP-44
> > > (internal-API), AIP-67 (multi-team) are more of those.
> > >
> > > So - what I think we might want to do as a community:
> > >
> > > * start working on Airflow 3 foundations (and decide what it means for
> > > our users and developer community). Decide what to keep, what to drop,
> > > what to redesign, assumptions to recreate.
> > >
> > > * explicitly split the initiatives/AIPs we have to target Airflow 2
> > > and Airflow 3 and treat them a bit differently in terms of
> > > future-proofness
> > >
> > > I would love to hear your thoughts on that (bracing for the storm of
> > > those).
> > >
> > > J.
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Kaxil Naik <ka...@gmail.com>]

Hello all,

I didn't anticipate reading an Airflow 3 email from a sunny beach in Nice,
France <https://en.wikipedia.org/wiki/Nice> -- I had a great time there
over the weekend, highly recommended :D

I say that because, as Vikram pointed out, some of us at Astronomer have
been polishing up the doc to propose Airflow 3 to the community in the
coming week. Such is the beauty of the open-source project that multiple
people (in the form of developers, committers, PMC members and various
Stakeholders) think the same. From the Astronomer front, Constance had been
championing a doc with Vikram & myself, with inputs from various other
committers & users, to have a good blend of different perspectives --
Product, PMC member & Industry leaders that cover several areas from
User-facing pain points, Industry trends in the Orchestration space,
Innovation in AI & ML space & opportunities as well as maintainability of
the current codebase. We would love to share it this week that goes into
some of the details to share our perspective on Why Airflow 3.0 & Why now
etc.

I would like to reiterate my statement from last year's panel session at
the Airflow Summit with Marc, Jarek & Pierre
<https://airflowsummit.org/sessions/2023/panels/panel-faces-airflow/>: "We,
as the Airflow project, have maintained a great balance of Innovation &
Stability", and I truly believe in that and is clearly visible in the
number of downloads and Airflow's popularity as the leader in the Workflow
Orchestration space. Our industry is rapidly evolving, especially in the
Data, AI & ML space. The role and expectations of the Data Orchestrator
(more specialized than a generic Workflow Orchestrator) are also evolving
as it is more and more utilized for Business critical applications than
just powering dashboards. So, IMO, we must continue catering to those
use-cases and innovate to aid the new use cases. Some of us from Astronomer
would like to create AIPs around these new use-cases LLM/Gen-AI, Data
Awareness, and some of the other things discussed above in the coming weeks
to receive feedback from everyone.

Apart from the new use cases around Data, AI & ML, balancing them with
resolving user pain points with things like DAG Versioning, a more modern
and extensible UI, lack of permissions with Airflow CLI, simplifying the
first-user Learning Curve etc while cleaning up Tech-debt (example we have
around 100+ deprecations in our code-base), a more performant scheduler
like the async SqlAlchemy & Scheduler discussions on the mailing list,
dropping dependency on FAB, rethinking provider/core separation -- both for
users & developers -- will make for a powerful 3.0 release that users will
want to update and will provide a cleaner code-base for the contributors to
build new foundational pieces.

Needless to say, similar to Airflow 2.0, we will have to provide our users
with utilities like the Airflow Upgrade check
<https://airflow.apache.org/docs/apache-airflow/stable/howto/upgrading-from-1-10/upgrade-check.html>
script
& other tools and docs to ease the migration.

Regarding the proposal to move the discussion to Confluence: In my opinion,
Confluence is a good place once things become more concrete and defined,
and we are looking for feedback. For a thing as big as Airflow 3, I would
humbly suggest the same route as what we did for Airflow 2 -- to have
a few recurring
Dev calls
<https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes> to
gather areas of interest and various upcoming AIPs from different
stakeholders and get aligned on Why Airflow 3? and Defining the high-level
scope
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158869992>.
This way, we can iterate much faster, and since all of the calls will be
recorded and summary notes will be added on Confluence (example
<https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes>) and
posted on the mailing list
<https://lists.apache.org/list?dev@airflow.apache.org:2020-9:dev%20call>,
we will have a written record of the important things. For any decision
points, we will bring things to the mailing list for Vote or Lazy
consensus. We could also re-use the Town Hall calls if needed. Once we have
good enough alignment and scope,  utilizing Confluence and mailing list on
the individual items & AIPs would be more valuable, in my opinion.

PS: It makes me very excited that we are discussing a new major phase of
Airflow. Looking forward to it.

Regards,
Kaxil

On Sun, 21 Apr 2024 at 17:21, Scheffler Jens (XC-AS/EAE-ADA-T)
<Je...@de.bosch.com.invalid> wrote:

> Hi Developers,
>
> TLDR Summary: I propose to move the discussion from a Email Replay-to-all
> chain to a discussion collection in
> https://cwiki.apache.org/confluence/x/hQv9EQ
>
> When I first saw this email from Jarek I was a bit surprised and actually
> the email was pulling me out of a kind of comfort zone, knowing what are
> the next steps. Naturally I was shocked a bit. So I decided to have a sleep
> over it. (Might be a bit shocking because Jarek dropped it 😃 especially I
> heard his position about a 3.0 before and that always sounded to me like a
> strong position... haha)
> After having a sleep-over the post I think it is valid to raise the
> discussion. Especially as we are going to a 10th feature-release which was
> also a cut-over from 1.x to 2.x. At some point every software product needs
> a re-factoring and cleanup. Structures are never perfect. But a lot of
> emotions and work are included with such a step. And a risk to fail and to
> lose a lot of users and force them to migrate (or have them run-away). So
> my current outcome is: We should carefully consider. But we need to
> consider.
>
> I believe the discussion will take a moment and focus - and a Reply-to-all
> chain will not be a good path as we will lose a lot of detail and focus and
> emails will create a lot of noise which is hard to follow. In a perfect
> non-distributed world I'd call you to a half-day visioning workshop in a
> room and focus on the whiteboard. Not possible with this level of
> distribution. Next option would be a (large) ~4h conference call which is
> hard to make in a time-zone matching the sleep cycle for all. Perfect would
> be if Summit would be close-by and plan a 1/2 day or full-day breakout for
> contributors on Day4 or so. But September is far far away.
>
> Therefore - to reduce amount of emails - I propose to start points, ideas,
> pain points etc. first on a Confluence page. Therefore I tried to start one
> page as starting points (contrary ideas welcome!) to have a place to
> collaborate and sketch. A virtual whiteboard would also be OK but I had
> none at my hands to share... (like Miro, Mural etc.). If we collect ideas,
> points etc. on this page we can have a rather short (2h) call with
> contributors in the next time to pitch and discuss the points and define
> follow-up steps to a plan, vote and conclusion.
>
> Proposed Confluence discussion page:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Discussion+and+Planning
>
> As a starting point I tried to import the both emails I saw in the thread
> into the page as starter. As it is a call to collaborate, please start
> editing and drop your points as well.
>
> Towards Jarek's mentioned trigger points:
> Actually the dropped AIP-68 and AIP-69 are something that in my view do
> NOT require Airflow to get to 3.0. I would see them either "Tactical" or
> "just functional enhancements". AIP-68 is "just" a bit of sugar to UI and
> extensions to Plugin interface in my view. AIP-69 is basically building
> something on-top, based on the concept of Hybrid Executors. As long as we
> would assume AIP-69 does not need drastical changes, maybe only small
> adjustments in the core (but concept not elaborated yet). I see this mainly
> as "just another Executor" that should not need breaking changes. I did not
> want to drop these two AIP's to start a fundamental discussion but rather
> to bring-in a new feature each.
> The points as factors that are hard to achieve in Airflow 2.x world are
> rather the "Multi Tenancy/Team" and "Dag Versioning" which in my eyes might
> be able to move faster with a 3.0.
>
> P.S.: I do not get the point (yet?) Why GenAI is a trigger point that
> forced structural breaking changes?
>
> Mit freundlichen Grüßen / Best regards
>
> Jens Scheffler
>
> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen |
> GERMANY | www.bosch.com
> Tel. +49 711 811-91508 | Mobil +49 160 90417410 |
> Jens.Scheffler@de.bosch.com
>
> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> Forschner,
> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
>
> -----Original Message-----
> From: Vikram Koka <vi...@astronomer.io.INVALID>
> Sent: Saturday, April 20, 2024 6:23 PM
> To: dev@airflow.apache.org
> Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs
> strategic (Airflow 3) approach
>
> A wonderful and exciting Saturday morning discussion!
> Thank you Jarek for bringing the offline conversations into the mailing
> list.
>
> I completely agree on the necessity of Airflow 3.
> I also agree that Gen AI is the trigger i.e. the answer to "Why now"?
>
> Having been thinking about this for a while from a strategic perspective,
> as opposed to the tactical perspective of the bi-weekly and monthly
> releases, I believe that our thinking as you articulated should have a
> clear understanding of strategic vs. tactical, but I don't believe our
> execution needs to necessarily be either or, but can actually be blended.
>
> With that said,  I believe that there are the following four buckets that
> we should use as a framework for Airflow 3.
>
> 1. Gen AI / LLM support
> 2. Airflow User Improvements
> 3. Easy adoption of Airflow by new users 4. Integration improvements /
> Provider maintainability
>
> Describing them in more detail below:
> 1. Gen AI / LLM support
> Reiterating the fact that this needs more work, I do believe this can be
> incremental to Airflow. As Astronomer, we have worked on the LLM Providers
> which we contributed to Airflow late last year. But clearly, there is so
> more to do, both from building awareness of the patterns / templates to
> use, as well as patterns to support in Airflow to make these easier to use
> and adopt.
>
> 2. Airflow User Improvements
> Clearly features and improvements desired by the Community are important
> to continue to work on to make Airflow more approachable. The top two
> features which leap to mind for me here are:
> 2.1 DAG Versioning - the most requested feature in the Airflow User Survey,
> 2.2 Modern UI - also comes up a lot
> 2.3 Different DAG distribution processes
> 2.4 Different execution mechanisms
> I know there are many more which I don't currently recall.
>
> 3. Airflow adoption
> We have discussed this many times, but we absolutely need to make the
> individual first-time adoption of Airflow better.
> I think the most common term I recall here is the notion of "Airflow
> Standalone", but whatever the term may be, an ultra quick, simple install
> of Airflow and the getting started experience is something we owe our
> community.
>
> 4. Integration / Providers
> The changes we made as part of Airflow 2.0 to split the Core Airflow
> releases from the Provider releases was clearly a good choice and made a
> huge impact. However, the integration maintainability balanced with growth
> still seems like it could use a significant set of improvements. Elad and I
> spoke about this a couple of days ago as well and I don't have a clear set
> of next steps here, but definitely worth exploring.
>
> Some of us at Astronomer have been discussing this quite a bit and
> planning on bringing a more polished draft to the community, but an initial
> discussion on a Saturday is fun as well :). We will definitely share our
> Airflow 3 proposal as a document with the community within the next week,
> as a request for comment.
>
>
>
> On Sat, Apr 20, 2024 at 1:50 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hello here,
> >
> > I have been thinking a lot recently and discussing with some people
> > and I am more and more convinced it's about the time we - as a
> > community - should start doing changes considering "Airflow 2" current
> and "Airflow 3" future.
> >
> >
> > *TL;DR: I think we should seriously start work on Airflow 3 and decide
> > what it means for our AIPs  - to treat some of them as more "tactical"
> > - things that should go into Airflow 2 and some "strategic" ones -
> > being foundational for Airflow 3 - with different goals and criteria.*
> >
> > A lot of us already think that way and a lot of us have already talked
> > about it for quite some time, so you should treat my mail mostly as a
> > little trigger "let's start publicly discussing what it might mean for
> > us and our community and let's make it clear about the target of the
> > initiatives we do".
> >
> > Some might be surprised it comes from me as I've been often saying "no
> > Airflow 3 without a good reason" or "possibly we will have no Airflow
> > 3", but I think (and a number of people I spoke to have similar
> > opinion) we have plenty of reasons to make some bold moves now.
> >
> > Over the last 4 years since Airflow 2 was out, a lot has changed and
> > we have a number of different needs that current Airflow 2 cannot
> > **really** do well
> >
> > - LLM/Gen-AI mainly as the important trigger
> > - Cloud Native is the "way to go"
> > - need to submit DAGs in other ways than dropping them to a shared DAG
> > folder.
> > - local testing and fast iteration on developing pipelines.
> > - ability to run tasks with workflow with "affinity" so that they can
> > share inputs/outputs in shared CPU/GPU memory
> > - ability to integrate seamlessly with other workflow engines - making
> > Airflow a "workflow of workflows
> > - probably way more
> > - all that while keeping a lot of the strengths of Airflow 2 - such as
> > continuing to have the option of using the many thousands of operators
> > with
> > 90+ providers.
> >
> > All those above - we could implement better if we get rid of a number
> > of the implicit or explicit luggage we have in Airflow 2. I think the
> > last two proposals from Jens: AIP-68 and AIP-69 reflect very much that
> > - both  would have been much easier and straightforward if we got
> > Airflow 3 re-designed basically at a drawing board with boldly
> > dropping some Airflow 3 assumptions.
> > And if we implemented core airflow 3 - taking the best part of what we
> > have now in Airflow 2, but generally dropping the luggage  in a new
> framework.
> >
> > And it won't be possible without breaking some fundamental assumptions
> > and making Airflow 3 quite heavily incompatible with Airflow 2
> >
> > From "my" camp - dropping the need of having the 700+ dependencies for
> > Airflow + all providers in a single Python interpreter, dropinnig
> > dependency on Flask/Plugins/FAB would be a huge win on its own. Not
> > mentioning being able to split provider's development and contribution
> > from airflow core (while keeping the development of providers as well
> > and
> > contributions) - this has been highly requested.
> >
> > And I think we have a lot of people in our community who would be able
> > (and would love) to do it - I think a number of us (including myself)
> > are a bit burned out and tired of just maintaining things in Airflow
> > in a backwards-compatible way and would jump on the opportunity to
> > rebuilding Airflow.
> >
> > But - we of course cannot forget about Airflow 2 users. We do not want
> > to "stop the world" for them. We want to keep fixing things and adding
> > incremental changes - and those things do not necessarily super
> > "future-proof". They should help  to "keep the lights on" for a while
> > - which means that in a number of cases it could be "band-aid". AIP-44
> > (internal-API), AIP-67 (multi-team) are more of those.
> >
> > So - what I think we might want to do as a community:
> >
> > * start working on Airflow 3 foundations (and decide what it means for
> > our users and developer community). Decide what to keep, what to drop,
> > what to redesign, assumptions to recreate.
> >
> > * explicitly split the initiatives/AIPs we have to target Airflow 2
> > and Airflow 3 and treat them a bit differently in terms of
> > future-proofness
> >
> > I would love to hear your thoughts on that (bracing for the storm of
> > those).
> >
> > J.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>

RE: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

["Scheffler Jens (XC-AS/EAE-ADA-T)" <Je...@de.bosch.com.INVALID>]

Hi Developers,

TLDR Summary: I propose to move the discussion from a Email Replay-to-all chain to a discussion collection in https://cwiki.apache.org/confluence/x/hQv9EQ

When I first saw this email from Jarek I was a bit surprised and actually the email was pulling me out of a kind of comfort zone, knowing what are the next steps. Naturally I was shocked a bit. So I decided to have a sleep over it. (Might be a bit shocking because Jarek dropped it 😃 especially I heard his position about a 3.0 before and that always sounded to me like a strong position... haha)
After having a sleep-over the post I think it is valid to raise the discussion. Especially as we are going to a 10th feature-release which was also a cut-over from 1.x to 2.x. At some point every software product needs a re-factoring and cleanup. Structures are never perfect. But a lot of emotions and work are included with such a step. And a risk to fail and to lose a lot of users and force them to migrate (or have them run-away). So my current outcome is: We should carefully consider. But we need to consider.

I believe the discussion will take a moment and focus - and a Reply-to-all chain will not be a good path as we will lose a lot of detail and focus and emails will create a lot of noise which is hard to follow. In a perfect non-distributed world I'd call you to a half-day visioning workshop in a room and focus on the whiteboard. Not possible with this level of distribution. Next option would be a (large) ~4h conference call which is hard to make in a time-zone matching the sleep cycle for all. Perfect would be if Summit would be close-by and plan a 1/2 day or full-day breakout for contributors on Day4 or so. But September is far far away.

Therefore - to reduce amount of emails - I propose to start points, ideas, pain points etc. first on a Confluence page. Therefore I tried to start one page as starting points (contrary ideas welcome!) to have a place to collaborate and sketch. A virtual whiteboard would also be OK but I had none at my hands to share... (like Miro, Mural etc.). If we collect ideas, points etc. on this page we can have a rather short (2h) call with contributors in the next time to pitch and discuss the points and define follow-up steps to a plan, vote and conclusion.

Proposed Confluence discussion page:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Discussion+and+Planning

As a starting point I tried to import the both emails I saw in the thread into the page as starter. As it is a call to collaborate, please start editing and drop your points as well.

Towards Jarek's mentioned trigger points: 
Actually the dropped AIP-68 and AIP-69 are something that in my view do NOT require Airflow to get to 3.0. I would see them either "Tactical" or "just functional enhancements". AIP-68 is "just" a bit of sugar to UI and extensions to Plugin interface in my view. AIP-69 is basically building something on-top, based on the concept of Hybrid Executors. As long as we would assume AIP-69 does not need drastical changes, maybe only small adjustments in the core (but concept not elaborated yet). I see this mainly as "just another Executor" that should not need breaking changes. I did not want to drop these two AIP's to start a fundamental discussion but rather to bring-in a new feature each.
The points as factors that are hard to achieve in Airflow 2.x world are rather the "Multi Tenancy/Team" and "Dag Versioning" which in my eyes might be able to move faster with a 3.0.

P.S.: I do not get the point (yet?) Why GenAI is a trigger point that forced structural breaking changes?

Mit freundlichen Grüßen / Best regards

Jens Scheffler

Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T)
Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY | www.bosch.com
Tel. +49 711 811-91508 | Mobil +49 160 90417410 | Jens.Scheffler@de.bosch.com

Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; 
Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus Forschner, 
Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert

-----Original Message-----
From: Vikram Koka <vi...@astronomer.io.INVALID> 
Sent: Saturday, April 20, 2024 6:23 PM
To: dev@airflow.apache.org
Subject: Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

A wonderful and exciting Saturday morning discussion!
Thank you Jarek for bringing the offline conversations into the mailing list.

I completely agree on the necessity of Airflow 3.
I also agree that Gen AI is the trigger i.e. the answer to "Why now"?

Having been thinking about this for a while from a strategic perspective, as opposed to the tactical perspective of the bi-weekly and monthly releases, I believe that our thinking as you articulated should have a clear understanding of strategic vs. tactical, but I don't believe our execution needs to necessarily be either or, but can actually be blended.

With that said,  I believe that there are the following four buckets that we should use as a framework for Airflow 3.

1. Gen AI / LLM support
2. Airflow User Improvements
3. Easy adoption of Airflow by new users 4. Integration improvements / Provider maintainability

Describing them in more detail below:
1. Gen AI / LLM support
Reiterating the fact that this needs more work, I do believe this can be incremental to Airflow. As Astronomer, we have worked on the LLM Providers which we contributed to Airflow late last year. But clearly, there is so more to do, both from building awareness of the patterns / templates to use, as well as patterns to support in Airflow to make these easier to use and adopt.

2. Airflow User Improvements
Clearly features and improvements desired by the Community are important to continue to work on to make Airflow more approachable. The top two features which leap to mind for me here are:
2.1 DAG Versioning - the most requested feature in the Airflow User Survey,
2.2 Modern UI - also comes up a lot
2.3 Different DAG distribution processes
2.4 Different execution mechanisms
I know there are many more which I don't currently recall.

3. Airflow adoption
We have discussed this many times, but we absolutely need to make the individual first-time adoption of Airflow better.
I think the most common term I recall here is the notion of "Airflow Standalone", but whatever the term may be, an ultra quick, simple install of Airflow and the getting started experience is something we owe our community.

4. Integration / Providers
The changes we made as part of Airflow 2.0 to split the Core Airflow releases from the Provider releases was clearly a good choice and made a huge impact. However, the integration maintainability balanced with growth still seems like it could use a significant set of improvements. Elad and I spoke about this a couple of days ago as well and I don't have a clear set of next steps here, but definitely worth exploring.

Some of us at Astronomer have been discussing this quite a bit and planning on bringing a more polished draft to the community, but an initial discussion on a Saturday is fun as well :). We will definitely share our Airflow 3 proposal as a document with the community within the next week, as a request for comment.



On Sat, Apr 20, 2024 at 1:50 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hello here,
>
> I have been thinking a lot recently and discussing with some people 
> and I am more and more convinced it's about the time we - as a 
> community - should start doing changes considering "Airflow 2" current and "Airflow 3" future.
>
>
> *TL;DR: I think we should seriously start work on Airflow 3 and decide 
> what it means for our AIPs  - to treat some of them as more "tactical" 
> - things that should go into Airflow 2 and some "strategic" ones - 
> being foundational for Airflow 3 - with different goals and criteria.*
>
> A lot of us already think that way and a lot of us have already talked 
> about it for quite some time, so you should treat my mail mostly as a 
> little trigger "let's start publicly discussing what it might mean for 
> us and our community and let's make it clear about the target of the 
> initiatives we do".
>
> Some might be surprised it comes from me as I've been often saying "no 
> Airflow 3 without a good reason" or "possibly we will have no Airflow 
> 3", but I think (and a number of people I spoke to have similar 
> opinion) we have plenty of reasons to make some bold moves now.
>
> Over the last 4 years since Airflow 2 was out, a lot has changed and 
> we have a number of different needs that current Airflow 2 cannot 
> **really** do well
>
> - LLM/Gen-AI mainly as the important trigger
> - Cloud Native is the "way to go"
> - need to submit DAGs in other ways than dropping them to a shared DAG 
> folder.
> - local testing and fast iteration on developing pipelines.
> - ability to run tasks with workflow with "affinity" so that they can 
> share inputs/outputs in shared CPU/GPU memory
> - ability to integrate seamlessly with other workflow engines - making 
> Airflow a "workflow of workflows
> - probably way more
> - all that while keeping a lot of the strengths of Airflow 2 - such as 
> continuing to have the option of using the many thousands of operators 
> with
> 90+ providers.
>
> All those above - we could implement better if we get rid of a number 
> of the implicit or explicit luggage we have in Airflow 2. I think the 
> last two proposals from Jens: AIP-68 and AIP-69 reflect very much that 
> - both  would have been much easier and straightforward if we got 
> Airflow 3 re-designed basically at a drawing board with boldly 
> dropping some Airflow 3 assumptions.
> And if we implemented core airflow 3 - taking the best part of what we 
> have now in Airflow 2, but generally dropping the luggage  in a new framework.
>
> And it won't be possible without breaking some fundamental assumptions 
> and making Airflow 3 quite heavily incompatible with Airflow 2
>
> From "my" camp - dropping the need of having the 700+ dependencies for 
> Airflow + all providers in a single Python interpreter, dropinnig 
> dependency on Flask/Plugins/FAB would be a huge win on its own. Not 
> mentioning being able to split provider's development and contribution 
> from airflow core (while keeping the development of providers as well 
> and
> contributions) - this has been highly requested.
>
> And I think we have a lot of people in our community who would be able 
> (and would love) to do it - I think a number of us (including myself) 
> are a bit burned out and tired of just maintaining things in Airflow 
> in a backwards-compatible way and would jump on the opportunity to 
> rebuilding Airflow.
>
> But - we of course cannot forget about Airflow 2 users. We do not want 
> to "stop the world" for them. We want to keep fixing things and adding 
> incremental changes - and those things do not necessarily super 
> "future-proof". They should help  to "keep the lights on" for a while 
> - which means that in a number of cases it could be "band-aid". AIP-44 
> (internal-API), AIP-67 (multi-team) are more of those.
>
> So - what I think we might want to do as a community:
>
> * start working on Airflow 3 foundations (and decide what it means for 
> our users and developer community). Decide what to keep, what to drop, 
> what to redesign, assumptions to recreate.
>
> * explicitly split the initiatives/AIPs we have to target Airflow 2 
> and Airflow 3 and treat them a bit differently in terms of 
> future-proofness
>
> I would love to hear your thoughts on that (bracing for the storm of 
> those).
>
> J.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

[Vikram Koka <vi...@astronomer.io.INVALID>]

A wonderful and exciting Saturday morning discussion!
Thank you Jarek for bringing the offline conversations into the mailing
list.

I completely agree on the necessity of Airflow 3.
I also agree that Gen AI is the trigger i.e. the answer to "Why now"?

Having been thinking about this for a while from a strategic perspective,
as opposed to the
tactical perspective of the bi-weekly and monthly releases, I believe that
our thinking as you articulated should have a
clear understanding of strategic vs. tactical, but I don't believe our
execution needs to necessarily be either or, but can actually
be blended.

With that said,  I believe that there are the following four buckets that
we should use as a framework for Airflow 3.

1. Gen AI / LLM support
2. Airflow User Improvements
3. Easy adoption of Airflow by new users
4. Integration improvements / Provider maintainability

Describing them in more detail below:
1. Gen AI / LLM support
Reiterating the fact that this needs more work, I do believe this can be
incremental to Airflow. As Astronomer,
we have worked on the LLM Providers which we contributed to Airflow late
last year. But clearly, there is so more to do,
both from building awareness of the patterns / templates to use, as well as
patterns to support in Airflow to make these
easier to use and adopt.

2. Airflow User Improvements
Clearly features and improvements desired by the Community are important to
continue to work on to make Airflow more approachable. The top two features
which leap to mind for me here are:
2.1 DAG Versioning - the most requested feature in the Airflow User Survey,
2.2 Modern UI - also comes up a lot
2.3 Different DAG distribution processes
2.4 Different execution mechanisms
I know there are many more which I don't currently recall.

3. Airflow adoption
We have discussed this many times, but we absolutely need to make the
individual first-time adoption of Airflow better.
I think the most common term I recall here is the notion of "Airflow
Standalone", but whatever the term may be, an
ultra quick, simple install of Airflow and the getting started experience
is something we owe our community.

4. Integration / Providers
The changes we made as part of Airflow 2.0 to split the Core Airflow
releases from the Provider releases was clearly
a good choice and made a huge impact. However, the integration
maintainability balanced with growth still seems like it could
use a significant set of improvements. Elad and I spoke about this a couple
of days ago as well and I don't have a clear
set of next steps here, but definitely worth exploring.

Some of us at Astronomer have been discussing this quite a bit and planning
on bringing a more polished draft to the community, but an initial
discussion on a Saturday is fun as well :). We will definitely share our
Airflow 3 proposal as a document with the community within the next week,
as a request for comment.



On Sat, Apr 20, 2024 at 1:50 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hello here,
>
> I have been thinking a lot recently and discussing with some people and I
> am more and more convinced it's about the time we - as a community - should
> start doing changes considering "Airflow 2" current and "Airflow 3" future.
>
>
> *TL;DR: I think we should seriously start work on Airflow 3 and decide what
> it means for our AIPs  - to treat some of them as more "tactical" - things
> that should go into Airflow 2 and some "strategic" ones - being
> foundational for Airflow 3 - with different goals and criteria.*
>
> A lot of us already think that way and a lot of us have already talked
> about it for quite some time, so you should treat my mail mostly as a
> little trigger "let's start publicly discussing what it might mean for us
> and our community and let's make it clear about the target of the
> initiatives we do".
>
> Some might be surprised it comes from me as I've been often saying "no
> Airflow 3 without a good reason" or "possibly we will have no Airflow 3",
> but I think (and a number of people I spoke to have similar opinion) we
> have plenty of reasons to make some bold moves now.
>
> Over the last 4 years since Airflow 2 was out, a lot has changed and we
> have a number of different needs that current Airflow 2 cannot **really**
> do well
>
> - LLM/Gen-AI mainly as the important trigger
> - Cloud Native is the "way to go"
> - need to submit DAGs in other ways than dropping them to a shared DAG
> folder.
> - local testing and fast iteration on developing pipelines.
> - ability to run tasks with workflow with "affinity" so that they can share
> inputs/outputs in shared CPU/GPU memory
> - ability to integrate seamlessly with other workflow engines - making
> Airflow a "workflow of workflows
> - probably way more
> - all that while keeping a lot of the strengths of Airflow 2 - such as
> continuing to have the option of using the many thousands of operators with
> 90+ providers.
>
> All those above - we could implement better if we get rid of a number of
> the implicit or explicit luggage we have in Airflow 2. I think the last two
> proposals from Jens: AIP-68 and AIP-69 reflect very much that - both  would
> have been much easier and straightforward if we got Airflow 3 re-designed
> basically at a drawing board with boldly dropping some Airflow 3
> assumptions.
> And if we implemented core airflow 3 - taking the best part of what we have
> now in Airflow 2, but generally dropping the luggage  in a new framework.
>
> And it won't be possible without breaking some fundamental assumptions and
> making Airflow 3 quite heavily incompatible with Airflow 2
>
> From "my" camp - dropping the need of having the 700+ dependencies for
> Airflow + all providers in a single Python interpreter, dropinnig
> dependency on Flask/Plugins/FAB would be a huge win on its own. Not
> mentioning being able to split provider's development and contribution from
> airflow core (while keeping the development of providers as well and
> contributions) - this has been highly requested.
>
> And I think we have a lot of people in our community who would be able (and
> would love) to do it - I think a number of us (including myself) are a bit
> burned out and tired of just maintaining things in Airflow in a
> backwards-compatible way and would jump on the opportunity to
> rebuilding Airflow.
>
> But - we of course cannot forget about Airflow 2 users. We do not want to
> "stop the world" for them. We want to keep fixing things and adding
> incremental changes - and those things do not necessarily super
> "future-proof". They should help  to "keep the lights on" for a while -
> which means that in a number of cases it could be "band-aid". AIP-44
> (internal-API), AIP-67 (multi-team) are more of those.
>
> So - what I think we might want to do as a community:
>
> * start working on Airflow 3 foundations (and decide what it means for our
> users and developer community). Decide what to keep, what to drop, what to
> redesign, assumptions to recreate.
>
> * explicitly split the initiatives/AIPs we have to target Airflow 2 and
> Airflow 3 and treat them a bit differently in terms of future-proofness
>
> I would love to hear your thoughts on that (bracing for the storm of
> those).
>
> J.
>