[Meeting Notes] Airflow 3.0 Dev call - 19 Sep 2024

[Kaxil Naik <ka...@gmail.com>]

Hey all,

I have updated our meeting notes document to summarize the discussion from our
last dev call.

Link:
https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-19September2024

To all those who attended, can you please double-check and add if
I have missed anything?

To all those who didn't join, if you disagree with anything in the Summary,
please voice your opinion.

I will send a separate email with the agenda for the next meeting but if
someone has an item for the agenda, please reach out to me or feel free to
add it to the doc
<https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda>
.

Regards,
Kaxil

------

Including the Summary here too (might break formatting):

*Catch-up on action items from last call*

   - Kaxil Naik showed the GitHub project boards
   <https://github.com/apache/airflow/projects?query=is%3Aopen> for some of
   the existing AIPs. Kaxil will follow up with all other AIP-leads to have
   the project board created.
   - Kaxil Naik discussed a proposal for how the intermediate milestones
   would be created to track progress. Kaxil will follow up with a draft with
   Vikram & Rajesh and then follow up with all the AIP leads to have the
   milestones defined. The general agreement was to have 3 intermediate
   milestones: Week of 24 Oct, 18 Nov and 16 Dec
   - Vikram Koka confirmed that Astronomer did not end up using Casbin
   itself but just the interfaces. Shubham Mehta mentioned Niko Oliveira is
   going to explore Casbin and Keycloak and start implementation in about a
   week. The Dex project <https://dexidp.io/> was suggested as yet another
   option by Mathew Wicks at the Airflow Summit. This was used by them in the
   Kubeflow project. His suggestion was to utilize it for Authentication only
   and continue doing Authorization on the Airflow side. Shubham will reach
   out to him.


*Discussions*

   - Split into Operators / Executors / Plugins (Jens)
      - The goal of the discussion was to split the providers into
      Executors, plugins, and what's needed for DAG Authors (i.e User
Code) like
      Operators and hooks. This way, we can limit the Python
dependencies needed
      for Webserver, Scheduler, Worker, etc. An example is an AWS provider that
      has an Executor as well as an Operator. If we split them, then Scheduler
      can use the Provider that has an Executor, and Worker (including
Triggerer
      & DAG Parser) can use the AWS provider that has an Operator but not an
      Executor. Airflow 3 is the right time to split this out if we
decide to go
      that way.
      - re External Operator Links: Some ideas were discussed, like the
      parser storing the links in the DB, which is then retrieved by the
      Webserver via an API, so the Webserver does not need to install
a provider
      just for Operator Links.
      - Jens Scheffler  will redraw the above diagram to separate out the
      Scheduler and the DAG Parser as the Scheduler does not run User Code, but
      the Parser does. As a follow-up to it, Jens will have a proposal around
      what the separation of providers would like to discuss on the next call
      and/or the dev mailing list. Vikram Koka and Kaxil Naik have
volunteered to
      work with Jens on it offline before the next call.
   - Airflow Standard Provider: Bash
   <https://github.com/apache/airflow/pull/42252> & Python
   <https://github.com/apache/airflow/pull/42081> Operator (Pavan Kumar)
      - Pavan brought up the concern that System tests will break if we
      change the example DAGs for Bash & Python from core to using Standard
      provider until it is released. This was because some of providers like
      Google use the Bash or Python Operators.
      - Ideally, it should work with Airflow 3 since the provider will be
      pre-installed (via required_deps) with Airflow core/SDK.
      - The team decided to take it offline. Reviews would be appreciated.
   - Potentially moving dev calls on Oct 3 to Oct 9 or Oct 10. And one on
   Oct 31 [*Diwali*] to Nov 1 (Kaxil Naik )
      - Decided to move 03 Oct 2024 call to10 Oct 2024  and move the 01 Nov
      2024  to 07 Nov 2024