[Meeting Notes] Airflow 3.0 dev call - 7 Nov 2024

[Vikram Koka <vi...@astronomer.io.INVALID>]

Hey everyone,

I updated our meeting notes document in the Airflow wiki to capture the
notes from our dev call yesterday, the 7th of November. The link for those
notes is here
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.12>


To everyone who attended the meeting, please check the summary and add
anything that I may have missed.
For those who could not join, please let us know if you disagree with
anything discussed and agreed upon in the meeting. Also, please do ask
questions if something is unclear.
There's already an initial agenda for our next dev call, which is scheduled
for 21th Nov. If you would like something to be added to the proposed
agenda for that meeting, please add it here
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-(Proposed)Agenda.2>
or
let me know.

I am very excited by the progress on Airflow 3 to date and very much
looking forward to the next milestone in a couple of weeks.
Truly appreciate all the hard work being put into this by all the
contributors!

Best regards and talk to you all soon,
Vikram
--
Below is the summary from the call yesterday:


   - Follow-up on action items from the last call:
      - Update on AIP-79 with respect to what's left (Jed Cunningham,
      Vincent Beck, Jens Scheffler): Jed shared that it was unclear at
this time
      if AIP-79 was blocked on AIP-38 (new UI).
         - Jed to report back by the next dev call if he could magic back
         the older Fab provider, so that the old Fab provider could exist
         independent of the new Fab provider, so that we are not
blocked by the new
         UI work
         - The Connection form for the new UI is another item which needs
         to be done and is not blocked by anything else. Jens's intention is to
         wait  and build the Connection form based on the Trigger form
in the new
         UI. Though to be clear, no one actively owns the Connection
form work at
         this time, but Brent and Jens believe that this will
naturally follow as
         part of the AIP-38 work.
         - Clarity from Jed and Vincent that the new Fab provider is needed
         for the Auth manager and more and is not in conflict with the
work that Jed
         is proposing above.
      - Breaking changes update (Vikram Koka): Vikram walked the group
      through his thoughts around capturing all changes that users
need to worry
      about as part of upgrading to Airflow 3. Specifically to structure the
      newsfragments with a template. Vikram proposed an initial
template with the
      following elements:
         -

         DAG code changes
         -

            Import changes
            -

            Other DAG code changes
            -

            e.g. SubDAGs - please note if it was already marked as
            “deprecated” or “experimental”
            -

         Stable API changes: both invocation and result
         -

         CLI changes:
         -

         Airflow configuration changes
         -

            Airflow.cfg changes
            -

            Integration changes
            -

         Behavior changes: Example Backfill or Execution date changes
         -

         Plugin changes
         - There was immediate agreement within the group for the above and
      the desire for a concrete template. Vikram to publish the above
and then it
      could be added as a pre-commit. Vikram to also create an initial
Upgrade /
      Change overview page, which would have example results based on
some of the
      initial newsfragments (around 10).
   - Development updates and presentations:
   - Airflow 3 workstreams update (Vikram Koka): Vikram walked the group
      through the updates to the Airflow 3 workstreams
      <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+-+Workstreams>
       and Airflow 3 dev milestones
      <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3.0+Development+Milestones>
pages
      to reflect the high level status of the Airflow 3 project as
well as finer
      level monthly details. Positive feedback from Rajesh and the rest of the
      team on the overall summarization in the Workstreams page, with the more
      granular detail in the monthly dev milestones page. Vikram reiterated the
      ask for more help on the AIP-38 UI work. Brent said that we are now ready
      for others to pickup work now, since the back-end APIs are ready to use.
      Brent to create a Slack sub-channel for UI development specifically for
      AIP-38 work.
   - Discussion topics:
   - Update other candidates from the Workstreams list:
         - Remove SLAs and AIP-57 Refactor SLA Feature - obsoleted by
         AIP-86 Deadline alerts?  (Dennis Ferruzzi) Dennis confirmed
that this entry
         was now obsoleted by AIP-86 Deadline Alerts.
         - Remove poke, reschedule mode from Sensors? Kaxil Naik said that
         there was a discussion in one of the early dev calls and that
he felt that
         this should be removed to really simplify the code and choices for the
         users. Daniel Standish felt that there was a difference
between support in
         Operators vs. Airflow Core. He felt that the support in
Operators was very
         confusing.  Shubham Mehta felt that this capability was
useful for very
         long running tasks (greater than 12 hours) in the past, but
wasn't sure if
         it was still necessary after Airflow 2.8, and would get back
to the team
         about the same. Jed Cunningham felt that it would be much
simpler to take
         it out. There was a good discussion with multiple members of
the team and
         both Igor Kholopov and Shubham Mehta confirmed that they didn't have a
         strict "no" at this time, but would need to confirm. After
the discussion,
         the action item was for Kaxil Naik to start an email
discussion around this
         in the dev list.
         - Inspect & Simplify Airflow Configurations: Shubham Mehta and
         Dennis Ferruzzi confirmed that we should just defer this to
post Airflow
         3.0.
      - OTEL integration into Airflow Scheduler (Daniel Standish):
         - There was a long discussion about the task log reading
         introduction into the very performance intensive Scheduler
loop and the
         challenges therein.
         - Kaxil Naik to start an async discussion around what changes need
         to be done to either refactor or revert these changes for OTEL traces.
      - Action items for next dev call:
      - Jed Cunningham to confirm back regarding the older Fab provider
      changes for compatibility.
      - Vikram Koka to create a template for newsfragments for flagging
      upgrade issues. And to create an initial Airflow 3 upgrade considerations
      page based on some of the existing PRs.
      - Brent Bovenzi to create an Airflow 3 UI slack channel to facilitate
      collaboration on Airflow 3 UI dev ("airflow-3-ui") - DONE
      - Kaxil Naik to start email discussion regarding removal of Poke and
      reschedule mode from sensors.
      - Kaxil Naik to send an email about the issues with OTEL integration
      for Traces into the Scheduler.