You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Vikram Koka <vi...@astronomer.io> on 2020/09/29 17:11:43 UTC

[Meeting Notes] Airflow 2.0 Dev Call #6 - 28 Sep 2020

Hi all,

I am sending this out since Kaxil is juggling other obligations. I have
updated our meeting notes document to summarize the discussion from
our dev call for Airflow 2.0 yesterday.

Thank you all who joined the call.

*Doc Link*:
https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#6:28Sep2020
<https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%236:28Sep2020>

To all those who attended, can you please double-check and add if I have
missed anything?
To all those who didn't join, if you disagree to anything in
the Summary please voice your opinion.

Also please let me know if someone wants to include an item in Next call's
Agenda.
Including the Summary here too (might potentially break formatting):
*Key Decisions*

   - *Scheduler HA*
      - *Locking discussion*: Document how Scheduler HA would work with a
      HA database configuration, such as an Active / Passive database
      configuration with multiple read replicas, but a single writer.
      - Add MySQL 8 to the our CI pipeline
      https://github.com/apache/airflow/issues/11164
      - *Progress update*: Ash said that the unit tests should be green by
      tomorrow. One thing left is the change_state_for_to_without_dagrun -
      currently called for every time (expensive), will change to call every 30
      seconds. PR is feature complete after that is done and will no longer be
      "in draft". Will need to move first functionally complete build to next
      week instead of current week, to give time for all unit tests to pass and
      for reviews.
      - *Benchmarks*: Will be run based on this branch, since the current
      benchmarks are based on an initial draft branch before rebase
with master.


   - *API*
      - *Task Instance Endpoint:* Only thing open. Kaxil to speak with
      Kamil to get status.
      - *Existing permissions map to UI*: WIP PRs. Feedback requested by
      James on this PR https://github.com/apache/airflow/pull/11158/
      - *Migrations: *Is there a different migration process needed? Can be
      run as a standard alembic migration.
      - *Clients:* QP said that the GO client based on the REST API is
      already completed and he is using it as part of his airflow-terraform
      module.
   - *UI Improvements*
      - *New HomePage:* Ryan has updated the UX (look) and shared on slack.
      Split Actions and Buttons, using Google Material Fonts. Feedback from
      meeting was very positive. PR to be final tomorrow.
   - *Splitting Providers Package*
      - *Separate provider packages from core: *Need some help to get this
      wrapped up. https://github.com/apache/airflow/issues/11163
      - *Build optimization*: This would really help speed-up builds by
      only running tests for changed providers
      https://github.com/apache/airflow/issues/10507
      - *SemVer*: Kaxil to send email to dev list confirming the decision
      in the meeting about using SemVer (lazy-consensus)
   - *Backport Providers in next 1.10.x release*
      - *EmailOperator: *Location has changed (moved to core)
      - *CNCF.Kubernetes providers*: Need to be checked if these can be
      released. *Insert github issue link created by Kamil for core
      operators*
   - *Timeline update:*
      - *Functionally complete build: *Will be on the week of Oct 5th,
      instead of this week as described above in the Scheduler HA notes.


The timeline on the main 2.0 planning page has also been updated to reflect
the above.
*Doc Link*:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning


Best regards,

Vikram

Re: [Meeting Notes] Airflow 2.0 Dev Call #6 - 28 Sep 2020

Posted by Jarek Potiuk <Ja...@polidea.com>.
Just an update from my side regarding planning:  I had some planning
meeting for the work for the next week and the only thing I will be able to
do (I plan it for tomorrow) is to release backport packages.

* Release a 2nd wave of Backport packages #10014
https://github.com/apache/airflow/issues/10014

I will not have time for this - but I believe Kaxil/Ash are working on it:
* Enable MySQL 8 CI jobs #11164
https://github.com/apache/airflow/issues/11164

I also plan to finish next week the review of HA scheduler to the point
that I will be comfortable with merging it. I saw the results posted by
Kaxiil, they look super promising and I think once we finish the review and
the CI works for MySQL 8, I am cool with merging it for people to start
testing.

The other things are up for grabs to anyone who would like to help - those
are not strong prerequisites for 2.0, Alpha release - they are mostly
optimizations or splitting the providers which can be released as part of
further beta releases, so they are non-blocking.

* [OPTIMISATION] Selective builds for different types of tests #10507
https://github.com/apache/airflow/issues/10507
* Fully separate provider packages from the Airflow core (AIP-8)
https://github.com/apache/airflow/issues/11163

J.




On Tue, Sep 29, 2020 at 7:12 PM Vikram Koka <vi...@astronomer.io> wrote:

> Hi all,
>
> I am sending this out since Kaxil is juggling other obligations. I have
> updated our meeting notes document to summarize the discussion from
> our dev call for Airflow 2.0 yesterday.
>
> Thank you all who joined the call.
>
> *Doc Link*:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#6:28Sep2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%236:28Sep2020>
>
> To all those who attended, can you please double-check and add if I have
> missed anything?
> To all those who didn't join, if you disagree to anything in
> the Summary please voice your opinion.
>
> Also please let me know if someone wants to include an item in Next call's
> Agenda.
> Including the Summary here too (might potentially break formatting):
> *Key Decisions*
>
>    - *Scheduler HA*
>       - *Locking discussion*: Document how Scheduler HA would work with a
>       HA database configuration, such as an Active / Passive database
>       configuration with multiple read replicas, but a single writer.
>       - Add MySQL 8 to the our CI pipeline
>       https://github.com/apache/airflow/issues/11164
>       - *Progress update*: Ash said that the unit tests should be green
>       by tomorrow. One thing left is the change_state_for_to_without_dagrun -
>       currently called for every time (expensive), will change to call every 30
>       seconds. PR is feature complete after that is done and will no longer be
>       "in draft". Will need to move first functionally complete build to next
>       week instead of current week, to give time for all unit tests to pass and
>       for reviews.
>       - *Benchmarks*: Will be run based on this branch, since the current
>       benchmarks are based on an initial draft branch before rebase with master.
>
>
>    - *API*
>       - *Task Instance Endpoint:* Only thing open. Kaxil to speak with
>       Kamil to get status.
>       - *Existing permissions map to UI*: WIP PRs. Feedback requested by
>       James on this PR https://github.com/apache/airflow/pull/11158/
>       - *Migrations: *Is there a different migration process needed? Can
>       be run as a standard alembic migration.
>       - *Clients:* QP said that the GO client based on the REST API is
>       already completed and he is using it as part of his airflow-terraform
>       module.
>    - *UI Improvements*
>       - *New HomePage:* Ryan has updated the UX (look) and shared on
>       slack. Split Actions and Buttons, using Google Material Fonts. Feedback
>       from meeting was very positive. PR to be final tomorrow.
>    - *Splitting Providers Package*
>       - *Separate provider packages from core: *Need some help to get
>       this wrapped up. https://github.com/apache/airflow/issues/11163
>       - *Build optimization*: This would really help speed-up builds by
>       only running tests for changed providers
>       https://github.com/apache/airflow/issues/10507
>       - *SemVer*: Kaxil to send email to dev list confirming the decision
>       in the meeting about using SemVer (lazy-consensus)
>    - *Backport Providers in next 1.10.x release*
>       - *EmailOperator: *Location has changed (moved to core)
>       - *CNCF.Kubernetes providers*: Need to be checked if these can be
>       released. *Insert github issue link created by Kamil for core
>       operators*
>    - *Timeline update:*
>       - *Functionally complete build: *Will be on the week of Oct 5th,
>       instead of this week as described above in the Scheduler HA notes.
>
>
> The timeline on the main 2.0 planning page has also been updated to
> reflect the above.
> *Doc Link*:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning
>
>
> Best regards,
>
> Vikram
>
>

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>