Re: [Meeting Notes] Airflow 2.0 Dev Call #6 - 28 Sep 2020

[Jarek Potiuk <Ja...@polidea.com>]

Just an update from my side regarding planning:  I had some planning
meeting for the work for the next week and the only thing I will be able to
do (I plan it for tomorrow) is to release backport packages.

* Release a 2nd wave of Backport packages #10014
https://github.com/apache/airflow/issues/10014

I will not have time for this - but I believe Kaxil/Ash are working on it:
* Enable MySQL 8 CI jobs #11164
https://github.com/apache/airflow/issues/11164

I also plan to finish next week the review of HA scheduler to the point
that I will be comfortable with merging it. I saw the results posted by
Kaxiil, they look super promising and I think once we finish the review and
the CI works for MySQL 8, I am cool with merging it for people to start
testing.

The other things are up for grabs to anyone who would like to help - those
are not strong prerequisites for 2.0, Alpha release - they are mostly
optimizations or splitting the providers which can be released as part of
further beta releases, so they are non-blocking.

* [OPTIMISATION] Selective builds for different types of tests #10507
https://github.com/apache/airflow/issues/10507
* Fully separate provider packages from the Airflow core (AIP-8)
https://github.com/apache/airflow/issues/11163

J.




On Tue, Sep 29, 2020 at 7:12 PM Vikram Koka <vi...@astronomer.io> wrote:

> Hi all,
>
> I am sending this out since Kaxil is juggling other obligations. I have
> updated our meeting notes document to summarize the discussion from
> our dev call for Airflow 2.0 yesterday.
>
> Thank you all who joined the call.
>
> *Doc Link*:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#6:28Sep2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%236:28Sep2020>
>
> To all those who attended, can you please double-check and add if I have
> missed anything?
> To all those who didn't join, if you disagree to anything in
> the Summary please voice your opinion.
>
> Also please let me know if someone wants to include an item in Next call's
> Agenda.
> Including the Summary here too (might potentially break formatting):
> *Key Decisions*
>
>    - *Scheduler HA*
>       - *Locking discussion*: Document how Scheduler HA would work with a
>       HA database configuration, such as an Active / Passive database
>       configuration with multiple read replicas, but a single writer.
>       - Add MySQL 8 to the our CI pipeline
>       https://github.com/apache/airflow/issues/11164
>       - *Progress update*: Ash said that the unit tests should be green
>       by tomorrow. One thing left is the change_state_for_to_without_dagrun -
>       currently called for every time (expensive), will change to call every 30
>       seconds. PR is feature complete after that is done and will no longer be
>       "in draft". Will need to move first functionally complete build to next
>       week instead of current week, to give time for all unit tests to pass and
>       for reviews.
>       - *Benchmarks*: Will be run based on this branch, since the current
>       benchmarks are based on an initial draft branch before rebase with master.
>
>
>    - *API*
>       - *Task Instance Endpoint:* Only thing open. Kaxil to speak with
>       Kamil to get status.
>       - *Existing permissions map to UI*: WIP PRs. Feedback requested by
>       James on this PR https://github.com/apache/airflow/pull/11158/
>       - *Migrations: *Is there a different migration process needed? Can
>       be run as a standard alembic migration.
>       - *Clients:* QP said that the GO client based on the REST API is
>       already completed and he is using it as part of his airflow-terraform
>       module.
>    - *UI Improvements*
>       - *New HomePage:* Ryan has updated the UX (look) and shared on
>       slack. Split Actions and Buttons, using Google Material Fonts. Feedback
>       from meeting was very positive. PR to be final tomorrow.
>    - *Splitting Providers Package*
>       - *Separate provider packages from core: *Need some help to get
>       this wrapped up. https://github.com/apache/airflow/issues/11163
>       - *Build optimization*: This would really help speed-up builds by
>       only running tests for changed providers
>       https://github.com/apache/airflow/issues/10507
>       - *SemVer*: Kaxil to send email to dev list confirming the decision
>       in the meeting about using SemVer (lazy-consensus)
>    - *Backport Providers in next 1.10.x release*
>       - *EmailOperator: *Location has changed (moved to core)
>       - *CNCF.Kubernetes providers*: Need to be checked if these can be
>       released. *Insert github issue link created by Kamil for core
>       operators*
>    - *Timeline update:*
>       - *Functionally complete build: *Will be on the week of Oct 5th,
>       instead of this week as described above in the Scheduler HA notes.
>
>
> The timeline on the main 2.0 planning page has also been updated to
> reflect the above.
> *Doc Link*:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning
>
>
> Best regards,
>
> Vikram
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>