You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Vikram Koka <vi...@astronomer.io.INVALID> on 2025/07/02 17:19:19 UTC
[Meeting Notes] Airflow 3 dev call - 26 June 2025
Hey everyone,
Thank you to everyone who made the dev call last week and apologies for
the delay in getting these notes out. I updated our meeting notes document
in the Airflow wiki to capture the notes. The link for those notes is here
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.30>
To everyone who attended the meeting, please check the summary and add
anything that I may have missed. For those who could not join, please let
us know if you disagree with anything discussed and agreed upon in
the meeting. Also, please do ask questions if something is unclear.
Our next meeting is scheduled for tomorrow, the 3rd of July at the same
time. The agenda is pretty short for now:
- Development Updates
- CLI Update (Bugra)
Please let me know if you would like to add anything to the agenda
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-ProposedAgenda.11>
.
Best regards,
Vikram
--
Below is the summary from the call:
- Catch-up on action items from last call:
- Docs site update (Jarek)
- Jarek mentioned that there is an issue
<https://github.com/apache/airflow/issues/51455> created for the
work here.
- The docs build process is a lot better now because of the work
led by Pavan, but the actual result still needs work.
- Jarek also spoke with Kamil about the original work done on the
docs site and his availability to help.
- Update regarding internationalization impact on release process
(Shahar)
- Shahar said there isn't any change to the process from prior
discussions.
- The expectation is for the Release Manager to announce that we
will cut a new release and that serves as a notification for
the code owner
to finish translation by a particular date.
- The medium term intent is to publish translations outside of
"Core Airflow". The initial concept here is to have
translations released
similar to provider releases and not to be tied to the core Airflow
software release. This could be just a file release or even a
data update
to the database, to be figured out.
- There was a lot of enthusiasm from the team about how this
project has happened so quickly and all the community
involvement here.
- Development Updates and Presentations (Please keep the updates to
less than 10 minutes):
- UI Plugins Update (Brent, Pierre)
- Brent and Pierre did a quick demo of the UI plugins work and
shared the progress to date.
- This included the common abstraction for the UI plugin
regardless if the UI should be in a new tab or an iFrame rendering
- The team was enthusiastic about the great progress here.
- Deadline Alerts: Update on conclusion for database tradeoff (Dennis)
- Ramit shared that the development completion target for Deadline
alerts is the end of July
- Ramit also shared that right now when a DAG is created with a
deadline alert, the deadline would be created and stored in
the database
- The scheduler loop change work is upcoming and the database
change approach will be figured out as part of that.
- Brent asked about needing to sync on the API endpoints for the
same. Dennis to sync with Brent over the course of the next week.
- Discussion topics:
- Multi-team discussion from dev list - (Jarek)
- There was long spirited discussion around the multi-team effort.
Jarek opened by asking for comments.
- Vikram went first, saying that he wanted to make sure that we
were not trading off the simplicity of Airflow starter experience for
individual data engineers with the complexity needed for large
organizations wanting granular operational procedures. Vikram
asserted that
as an open source project, viral adoption has been the key to
Airflow's
success and that simplicity was important to maintain and in fact make
better.
- Jarek immediately agreed with that assertion and said that was a
key point in his desire to also simplify this AIP to
multi-team, from the
original, far more complex, multi-tenancy proposal.
- Vikram also said that he was surprised by the lack of reference
to AIP-82 Event driven scheduling, since this was such a commonly used
mechanism for segmenting Airflow deployments across teams
from a ownership
perspective, following classic distributed system principles.
- Jens also chimed in saying that we should perhaps articulate
this better in the context of how to set up Airflow for large
enterprises.
- There was a significant level of debate around the complexity
vs. simplicity tradeoff, with Rajesh stating this is probably the most
common requirement that MWAA hears from their customers. Niko
also chimed
in saying that we should be willing to accept some tradeoffs
to simplicity.
- There was a further discussion on what needs in priority order
needed to be addressed here, with Kaxil raising the point
that this raised
significant operational considerations and he wanted to make
sure that we
were not getting in over our skis in complexity with quota
management, etc.
- This ran out of time without an explicit decision in the meeting
itself.
- Jarek concluded by asking people to comment back in the AIP
soon, so that feedback could be addressed in writing.
- Call for actions ...ah.. review! Example Dags! (Jens)
- Jens raised a "call for action", asking people to help review
the work on the Example DAGs and to contribute
--
Vikram Koka
Chief Strategy Officer
Email: vikram@astronomer.io
<https://www.astronomer.io/>