[Meeting Notes] Airflow 3.0 Dev call - 25 July 2024

[Kaxil Naik <ka...@gmail.com>]

Hey all,

I have updated our meeting notes document to summarize the discussion
from our 25th
July dev call for Airflow 3.0.

Link:
https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-25July2024

To all those who attended, can you please double-check and add if
I have missed anything?

To all those who didn't join, if you disagree with anything in the Summary,
please voice your opinion.

I will send a separate email with the agenda for the next meeting on
8th August, but if someone has an item for the agenda, please reach out to
me or feel free to add it to the doc
<https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-(Proposed)Agenda>
.

Regards,
Kaxil

------

Including the Summary here too (might break formatting):

*Catch-up on action items from last call*

   - Kaxil created GitHub issues for workstream items with no owners; they
   have also been added as sub-items in this meta GitHub issue that tracks
   the 3.0 release <https://github.com/apache/airflow/issues/39593>. We had
   already had some interest from community members on some of these items,
   check individual issues for it.


*Discussions*
Branching Strategy:

   - As soon as 2.10.0rc1 is created, the Airflow main branch will become
   the Airflow 3 branch on *09 Aug 2024 *.
   - Providers will continue to be developed on the main branch since we
   have the framework to run provider tests against older Airflow versions.
   - Bugfixes: If there is a PR with a bugfix to main, the committers
   should ask the PR author to raise a PR against v2-10-test  branch too.
   The team agreed to create a mailing list thread to discuss different
   approaches offline. The thread has been initiated here
   <https://lists.apache.org/thread/gjlotg36lvp6qxzblk4fwv6wymof09h1>.


Which Python version should we support in Airflow 3?

   - The consensus was to keep things as-is i.e following the current
Python-support
   policy
   <https://github.com/apache/airflow?tab=readme-ov-file#support-for-python-and-kubernetes-versions>
where
   we would remove the support for a Python version when it is EOL.


Airflow 2 support policy including scope (feature vs bug fixes + security
only) & support period

   - We might release 2.11 and mark it as a bridge release to add
   compatibility shims or deprecation warnings similar to 1.10.15. But we will
   decide on it as and when such a need arises.
   - For providers, we will need to increase the support window for the
   minimum Airflow version from the current 6 months. We will continue Airflow
   2 provider support for a year or longer, exact period TBD. We will revisit
   this again in the coming months. If we cut 2.11 (or whatever the final 2.x
   version is), then after 6-12 months we could increase the minimum Airflow
   to 2.11. There were ideas around using a "compat" provider that might help
   with AF 2 and AF 3 stuff. This will be another thing we will discuss async
   at a later time to keep current focus on scope & AIPs.


*Workstream statuses for ones that does not have a vote started:*

   - AIP-57 Refactor SLA Feature (Shubham Mehta):
      - It was agreed that for now, we will mark it for removal in Airflow
      3.0 and adding it back in 3.1. We will assess again in the
coming dev calls
      if something changes. Elad has also offered to help if needed.
   - AIP-67 Multi-team deployment of Airflow components (Jarek Potiuk ):
      - Jarek is waiting to hear back on his answers to some of the doubts
      raised in the discussion thread and AIP. This might also be moved to
      Airflow 3.1 since it depends on AIP-72 and AIP-66 and the implementation
      hasn't been started on those yet.
   - AIP-68 Extended Plugin Interface for React Views (Jens
   Scheffler / Brent Bovenzi ):
      - As of today, the vote has been started on the mailing list (link
      <https://lists.apache.org/thread/x8k3khtw6nqbt88r9qlc4b0s1y4bwvf7>).
      During the meeting, a few things were discussed, including potentially
      renaming UI plugins to UI Extensions as the "Plugin" word has been
      overloaded. The other suggestion was to rename the AIP title to "New
      Interface" instead of "Extending the interface."
   - AIP-76 Asset Partitions (Tzu-ping Chung):
      - TP will send this out for VOTE in the next couple of days.
   - AIP-79 Replace Flask App Builder (Brent Bovenzi ):
      - Two big questions pending:
         1. Replacement for FAB Auth manager
         2. How does adding a FAB as Provider work? Can it allow users to
         make older plugins work?
      - Jed has been working on a POC on it. He is on vacation this week,
      but based on initial investigations, he was positive about the
progress. We
      will check it back in the next dev call when he is back.
      - The pending item is also to update the AIP to add the decoupling of
      Providers's Connections metadata & form from FAB.
   - AIP-80: Explicit Template Fields in Operator Arguments (Tzu-ping
   Chung ):
      - VOTE has been kicked off here
      <https://lists.apache.org/thread/s5g0zvjjxlpgnp718vs6n86qtlgzcgpw>.
   - Poll external Datasets to have event-based DAG scheduling (Vincent
   BECK / Shubham Mehta): Draft AIP
   <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow>
has
   been created as mentioned in this mailing list thread
   <https://lists.apache.org/thread/b7kl9sng7zn81dchn5nqom336ql5pnf2>.
   - Respect permissions in CLI: Buğra Öztürk
      - It now has an AIP-81: Enhanced Security in CLI via Integration of
      API
      <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-81%3A+Enhanced+Security+in+CLI+via+Integration+of+API>
and
      is awaiting feedback.
   - Returning DAG Results / Synchronous DAG Execution
   <https://docs.google.com/document/d/1CXis3Ovmxm3pt4moiBIpsajX7GmRpGbcEfwt_blkSLI/edit?usp=sharing>
   : Vikram Koka
      - The first proposal (Make Execution DAG non-unique) from the
      original doc is separated out into a different workstream led by TP.
      - Language support has been excluded from the scope for Airflow 3.0.
      For now, the only change considered would be to add the support to the
      existing Trigger API to return an async result which doesn't need an AIP.
      - The other changes for GenAI will now be implemented as a new
      provider.
   - Make Execution Date to be non-unique for a DAG: Tzu-ping Chung
      - In the next couple of days, a discussion thread with a draft AIP
      will be created by Tzu-ping Chung , followed by a vote.


   - Splitting providers from the mono-repo (Kaxil Naik)
      - Kaxil is not going to propose this anymore as he agrees that doing
      this now will increase the work and split the focus of folks. So we are
      going to NOT split providers from mono-repo to laser focus on Airflow 3
      feature work.
      - We might think about it at a later date after Airflow 3.0 is
      released.
   - AIP-49 OpenTelemetry Support for Apache Airflow
   <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-49+OpenTelemetry+Support+for+Apache+Airflow>
   :
      - The PR for the 2nd part
      <https://github.com/apache/airflow/pulls?q=is%3Apr+opentelemetry+is%3Aclosed>
got
      merged for 2.10 and the AIP is marked complete.
      - Howard Yoo 's comment: Initially we thought to collect logs in
      standard way (scraping from log files), but found a much better way to
      collect logs, by utilizing log api to get the task logs, and
then attaching
      them as 'span events' which would be much more useful. The extraction of
      Task Logs and attaching them as span events is thus included in
the AIP-49,
      thus it is not 'excluded', albeit not as a separate 'log' data as
      originally thought.
   - StatsD vs Open Telemetry
      - There was a discussion on whether we should mark StatsD as removed
      in Airflow 2.10 itself in the docs here
      <https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html>.
      The conclusion was that we should discuss this more as this has a wider
      impact including but not limited to the official Helm Chart which uses
      StatsD, the services providers (Astronomer, Composer, MWAA) and other
      long-time Airflow users. OTel also had limitation around character length
      limit and that you have to use tags with OTel but it didn't allow
      aggregating metrics with different tags etc. Shubham Mehta is going to
      check this out with his team and see if there are other concerns.


Triage Help needed: There are 360 open feature requests
<https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Akind%3Afeature+sort%3Aupdated-asc>.
Some we might want to consider and seek help for in 3.0, and others we
should close as not planned. Elad, Jens, Jarek, Vikram & Kaxil have
volunteered to look at those issues. Elad is also going to check with Dosu
team to see if we can re-label these old issues so they are more accurate.
Categorizing these issues into "areas" label was one of the next steps too
so that the volunteers can look at a particular area or some segment to
divide & conquer. Elad Kalif could you take the one on for issue-triage?