You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@apache.org> on 2020/09/25 19:08:59 UTC
[Meeting Notes] Airflow 2.0 Dev Call #5 - 21 Sep 2020
Hi all,
I have created a document to summarize the discussion from our dev call for
Airflow 2.0 this Monday. Apologies for the delay in publishing the Meeting
Notes.
Thank you all who joined the call.
*Doc Link*:
https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#5:21Sep2020
To all those who attended, can you please double-check and add if I have
missed anything?
To all those who didn't join, if you disagree to anything in
the Summary please voice your opinion.
Also please let me know if someone wants to include an item in Next call's
Agenda.
Including the Summary here too (might potentially break formatting):
*Key Decisions*
- *API*
- Progress:
- Project Board: https://github.com/apache/airflow/projects/1
- The issues labelled with "Enhancement" are not a requirement
for 2.0
- Endpoints:
- Task Instance Endpoint
<https://github.com/apache/airflow/pull/9597> is WIP, all the
other endpoints have been implemented.
- Permissions Model:
- PR <https://github.com/apache/airflow/pull/10594> has been
merged.
- The next piece of work to be done is migrating existing Views
to use resource-based permissions. (Github issue
<https://github.com/apache/airflow/issues/10469>). This is
mainly for standardizing the permissions model across API and UI.
- *Providers*
- Vote on AIP-8 took place on the mailing list
<https://lists.apache.org/thread.html/rcd63bbe62a618c4547bd00b1c1d14dc329cfe1c09e4795571be28cb3%40%3Cdev.airflow.apache.org%3E>
.
- There is an ongoing discussion on the same thread about SemVer vs
CalVer for the Providers package.
- The people involved on the call were *leaning towards SemVer* to
make a clear distinction about a breaking release. This will
potentially
increase the work on release managers but some automation
around releasing
(similar to backport providers) and automation around the
generation of the
changelog for the providers would make the effort less painful.
- *Version Per Provide: *Each Providers package would have a separate
versioning i.e. we might release "google-providers 3.1" and
"amazon-providers" 3.7 at the same time but the versioning for a
particular
provider will be independent of other providers.
- *DEV*
- Would be good to have a release policy on when we can deprecate a
feature, our release cadence. A good example is
https://docs.djangoproject.com/en/3.1/internals/release-process/#release-cadence
- *SubDag Deprecation*
- There is a mailing list thread
<https://lists.apache.org/thread.html/ra52746f9c8274469d343b5f0251199de776e75ab75ded6830886fb6a%40%3Cdev.airflow.apache.org%3E>
on
whether or not we want to deprecate SubDags in favor of Taskgroups, the
majority on the call agreed that we *should not deprecate the Subdags
yet* and wait till people have used TaskGroups and it has feature
parity with SubDags.
- However, we should *clearly recommend using TaskGroups compared to
SubDags* in our docs and state limitations of the SubDags.
- *Helm Chart Release*
- Deferred until 2.0 is out
- Will be available to use from the source code of Airflow on Github
but the first official release of the Helm chart will only happen after
Airflow 2.0
- *Docs*
- Mailing list thread
<https://lists.apache.org/thread.html/rc6331d0bf97d91dc88853c992513f4e886f113c1cff030876996859e%40%3Cdev.airflow.apache.org%3E>
to
get some feedback has been created and cross-posted across Slack and
Twitter. Once we have enough feedback, Kaxil will create Github
issues for
them so that anyone willing to help on it can start working on it.
- A separate section for Upgrading to 2.0 would be ideal, can be a
duplicate of Updating.md but with a better structure and more organized.
- *UI Changes*
- *Github Issue: *https://github.com/apache/airflow/issues/10953
- There are some proposals from Ryan for the UI changes for which he
has created some PRs (links below) and in the process of
creating few more.
- Task Instance Modal UX Enhancements · Issue #10944 ·
apache/airflow <https://github.com/apache/airflow/pull/10944>
- Replace JS package toggle w/ pure CSS solution #11035
<https://github.com/apache/airflow/pull/11035>
- Task Instance header/navigation pattern UX cleanup
<https://github.com/apache/airflow/pull/11089> – Suggestions /
VOTE needed here if anyone has strong opinions
- *Scheduler HA*
- *Reminder*: A draft PR for Scheduler HA
<https://github.com/apache/airflow/pull/10956> is available for
review. It would be good to get some more feedback from the
wider community
with their own DEV setup if possible.
- *Process*
- Any new PRs would continue to be merged until we complete the items
for 2.0 and release alphas.
- *NOTE: *The Timeline shown on the Planning page
<https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning>
will
be revisited every week on the Dev Call and updated if needed based on the
progress towards the major features of Airflow 2.0
Regards,
Kaxil
Re: [Meeting Notes] Airflow 2.0 Dev Call #5 - 21 Sep 2020
Posted by Kaxil Naik <ka...@gmail.com>.
Sure, yeah I will add this to today's agenda and send a reminder email for
today's meeting in a bit.
On Mon, Sep 28, 2020 at 3:43 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I would like to add one more topic for today. It's been raised when I
> spoke to my friend from GitLab (they solved similar problems) about the
> Locking architecture of the HA scheduler.
> It would be great to understand what kind of Database HA it will be OK
> with. I think this is quite important to add to the documentation when we
> merge it, but it might be good to talk about it today if we find the time.
> I think this is a really important aspect before we ask the users to test
> it.
> Just wanted to give the heads up before - in case we need some exploration
> before this we might talk about it at the next meeting as well.
>
> J.
>
>
> On Mon, Sep 28, 2020 at 12:56 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Hi Jarek,
>>
>> Thanks agree with MySQL 8 part, I will try to tackle that in a couple of
>> days and help out with the other PRs / issues too. The Selective
>> Optimisation one is a very interesting one too and will take a look at the
>> WIP PR you have already created.
>>
>> Regards,
>> Kaxil
>>
>>
>>
>> On Sat, Sep 26, 2020 at 10:13 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> Thanks, Kaxil, Good summary!
>>>
>>> Just a comment on progress here from my side. I have just updated the
>>> status of three issues that are relevant for our Monday discussion.
>>>
>>> * Enable MySQL 8 CI jobs #11164
>>> https://github.com/apache/airflow/issues/11164
>>>
>>> This one is a prerequisite IMHO to get HA Scheduler merged because we
>>> have a dependency on MySQL 8 for it but we do not test MySQL 8 on CI at all
>>> currently. I think this one should be implemented as part of the HA
>>> Scheduler PR merge preparation. I'd love to take that on, but I have no
>>> time to try it out even, but likely it is only adding the version and it
>>> should work, however, there is quite a risk that we will need some fixes at
>>> least in the tests and we for sure need to adapt Dockerfile to use MySQL 8
>>> client.
>>>
>>> It is also (not very strongly) depends on those three that are
>>> closely related. It would be best if those three are all completed because
>>> this will give us a chance to test the separation of packages and in case
>>> we are going Semver (which we all agree is a better approach), there is
>>> quite some work to implement SEMVER versioning for separate packages. This
>>> separation is also needed to massively speed up our CI builds and it will
>>> help us to tackle increased CI pressure when we add MySQL 8.
>>>
>>> * [OPTIMISATION] Selective builds for different types of tests #10507
>>> https://github.com/apache/airflow/issues/10507
>>> * Fully separate provider packages from the Airflow core (AIP-8)
>>> https://github.com/apache/airflow/issues/11163
>>> * Release a 2nd wave of Backport packages #10014
>>> https://github.com/apache/airflow/issues/10014
>>>
>>> Unfortunately, due to other obligations, I likely won't be able to
>>> complete any of those this week and I will have very little time this week
>>> in general, so any help on those is appreciated - if anyone would like to
>>> take any of those - I linked all the WIP PRs for those - so if there is
>>> anyone who would like to take it over - feel free.
>>>
>>> I will do my best to be able to take part in the meeting on Monday.
>>>
>>> J.
>>>
>>>
>>> On Fri, Sep 25, 2020 at 9:09 PM Kaxil Naik <ka...@apache.org> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have created a document to summarize the discussion from our dev call for
>>>> Airflow 2.0 this Monday. Apologies for the delay in publishing the Meeting
>>>> Notes.
>>>>
>>>> Thank you all who joined the call.
>>>>
>>>> *Doc Link*:
>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#5:21Sep2020
>>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%235:21Sep2020>
>>>>
>>>> To all those who attended, can you please double-check and add if I
>>>> have missed anything?
>>>>
>>>> To all those who didn't join, if you disagree to anything in
>>>> the Summary please voice your opinion.
>>>>
>>>> Also please let me know if someone wants to include an item in Next
>>>> call's Agenda.
>>>>
>>>> Including the Summary here too (might potentially break formatting):
>>>>
>>>> *Key Decisions*
>>>>
>>>> - *API*
>>>> - Progress:
>>>> - Project Board: https://github.com/apache/airflow/projects/1
>>>> - The issues labelled with "Enhancement" are not a
>>>> requirement for 2.0
>>>> - Endpoints:
>>>> - Task Instance Endpoint
>>>> <https://github.com/apache/airflow/pull/9597> is WIP, all
>>>> the other endpoints have been implemented.
>>>> - Permissions Model:
>>>> - PR <https://github.com/apache/airflow/pull/10594> has
>>>> been merged.
>>>> - The next piece of work to be done is migrating existing
>>>> Views to use resource-based permissions. (Github issue
>>>> <https://github.com/apache/airflow/issues/10469>). This is
>>>> mainly for standardizing the permissions model across API and UI.
>>>> - *Providers*
>>>> - Vote on AIP-8 took place on the mailing list
>>>> <https://lists.apache.org/thread.html/rcd63bbe62a618c4547bd00b1c1d14dc329cfe1c09e4795571be28cb3%40%3Cdev.airflow.apache.org%3E>
>>>> .
>>>> - There is an ongoing discussion on the same thread about SemVer
>>>> vs CalVer for the Providers package.
>>>> - The people involved on the call were *leaning towards
>>>> SemVer* to make a clear distinction about a breaking release.
>>>> This will potentially increase the work on release managers but some
>>>> automation around releasing (similar to backport providers) and automation
>>>> around the generation of the changelog for the providers would make the
>>>> effort less painful.
>>>> - *Version Per Provide: *Each Providers package would have a
>>>> separate versioning i.e. we might release "google-providers 3.1" and
>>>> "amazon-providers" 3.7 at the same time but the versioning for a particular
>>>> provider will be independent of other providers.
>>>> - *DEV*
>>>> - Would be good to have a release policy on when we can deprecate a
>>>> feature, our release cadence. A good example is
>>>> https://docs.djangoproject.com/en/3.1/internals/release-process/#release-cadence
>>>> - *SubDag Deprecation*
>>>> - There is a mailing list thread
>>>> <https://lists.apache.org/thread.html/ra52746f9c8274469d343b5f0251199de776e75ab75ded6830886fb6a%40%3Cdev.airflow.apache.org%3E> on
>>>> whether or not we want to deprecate SubDags in favor of Taskgroups, the
>>>> majority on the call agreed that we *should not deprecate the
>>>> Subdags yet* and wait till people have used TaskGroups and it
>>>> has feature parity with SubDags.
>>>> - However, we should *clearly recommend using TaskGroups
>>>> compared to SubDags* in our docs and state limitations of the
>>>> SubDags.
>>>> - *Helm Chart Release*
>>>> - Deferred until 2.0 is out
>>>> - Will be available to use from the source code of Airflow on
>>>> Github but the first official release of the Helm chart will only happen
>>>> after Airflow 2.0
>>>> - *Docs*
>>>> - Mailing list thread
>>>> <https://lists.apache.org/thread.html/rc6331d0bf97d91dc88853c992513f4e886f113c1cff030876996859e%40%3Cdev.airflow.apache.org%3E> to
>>>> get some feedback has been created and cross-posted across Slack and
>>>> Twitter. Once we have enough feedback, Kaxil will create Github issues for
>>>> them so that anyone willing to help on it can start working on it.
>>>> - A separate section for Upgrading to 2.0 would be ideal, can be
>>>> a duplicate of Updating.md but with a better structure and more organized.
>>>> - *UI Changes*
>>>> - *Github Issue: *https://github.com/apache/airflow/issues/10953
>>>> - There are some proposals from Ryan for the UI changes for
>>>> which he has created some PRs (links below) and in the process of creating
>>>> few more.
>>>> - Task Instance Modal UX Enhancements · Issue #10944 ·
>>>> apache/airflow <https://github.com/apache/airflow/pull/10944>
>>>> - Replace JS package toggle w/ pure CSS solution #11035
>>>> <https://github.com/apache/airflow/pull/11035>
>>>> - Task Instance header/navigation pattern UX cleanup
>>>> <https://github.com/apache/airflow/pull/11089> – Suggestions
>>>> / VOTE needed here if anyone has strong opinions
>>>> - *Scheduler HA*
>>>> - *Reminder*: A draft PR for Scheduler HA
>>>> <https://github.com/apache/airflow/pull/10956> is available for
>>>> review. It would be good to get some more feedback from the wider community
>>>> with their own DEV setup if possible.
>>>> - *Process*
>>>> - Any new PRs would continue to be merged until we complete the
>>>> items for 2.0 and release alphas.
>>>> - *NOTE: *The Timeline shown on the Planning page
>>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning> will
>>>> be revisited every week on the Dev Call and updated if needed based on the
>>>> progress towards the major features of Airflow 2.0
>>>>
>>>>
>>>> Regards,
>>>> Kaxil
>>>>
>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: [Meeting Notes] Airflow 2.0 Dev Call #5 - 21 Sep 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
I would like to add one more topic for today. It's been raised when I spoke
to my friend from GitLab (they solved similar problems) about the Locking
architecture of the HA scheduler.
It would be great to understand what kind of Database HA it will be OK
with. I think this is quite important to add to the documentation when we
merge it, but it might be good to talk about it today if we find the time.
I think this is a really important aspect before we ask the users to test
it.
Just wanted to give the heads up before - in case we need some exploration
before this we might talk about it at the next meeting as well.
J.
On Mon, Sep 28, 2020 at 12:56 PM Kaxil Naik <ka...@gmail.com> wrote:
> Hi Jarek,
>
> Thanks agree with MySQL 8 part, I will try to tackle that in a couple of
> days and help out with the other PRs / issues too. The Selective
> Optimisation one is a very interesting one too and will take a look at the
> WIP PR you have already created.
>
> Regards,
> Kaxil
>
>
>
> On Sat, Sep 26, 2020 at 10:13 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Thanks, Kaxil, Good summary!
>>
>> Just a comment on progress here from my side. I have just updated the
>> status of three issues that are relevant for our Monday discussion.
>>
>> * Enable MySQL 8 CI jobs #11164
>> https://github.com/apache/airflow/issues/11164
>>
>> This one is a prerequisite IMHO to get HA Scheduler merged because we
>> have a dependency on MySQL 8 for it but we do not test MySQL 8 on CI at all
>> currently. I think this one should be implemented as part of the HA
>> Scheduler PR merge preparation. I'd love to take that on, but I have no
>> time to try it out even, but likely it is only adding the version and it
>> should work, however, there is quite a risk that we will need some fixes at
>> least in the tests and we for sure need to adapt Dockerfile to use MySQL 8
>> client.
>>
>> It is also (not very strongly) depends on those three that are
>> closely related. It would be best if those three are all completed because
>> this will give us a chance to test the separation of packages and in case
>> we are going Semver (which we all agree is a better approach), there is
>> quite some work to implement SEMVER versioning for separate packages. This
>> separation is also needed to massively speed up our CI builds and it will
>> help us to tackle increased CI pressure when we add MySQL 8.
>>
>> * [OPTIMISATION] Selective builds for different types of tests #10507
>> https://github.com/apache/airflow/issues/10507
>> * Fully separate provider packages from the Airflow core (AIP-8)
>> https://github.com/apache/airflow/issues/11163
>> * Release a 2nd wave of Backport packages #10014
>> https://github.com/apache/airflow/issues/10014
>>
>> Unfortunately, due to other obligations, I likely won't be able to
>> complete any of those this week and I will have very little time this week
>> in general, so any help on those is appreciated - if anyone would like to
>> take any of those - I linked all the WIP PRs for those - so if there is
>> anyone who would like to take it over - feel free.
>>
>> I will do my best to be able to take part in the meeting on Monday.
>>
>> J.
>>
>>
>> On Fri, Sep 25, 2020 at 9:09 PM Kaxil Naik <ka...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> I have created a document to summarize the discussion from our dev call for
>>> Airflow 2.0 this Monday. Apologies for the delay in publishing the Meeting
>>> Notes.
>>>
>>> Thank you all who joined the call.
>>>
>>> *Doc Link*:
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#5:21Sep2020
>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%235:21Sep2020>
>>>
>>> To all those who attended, can you please double-check and add if I have
>>> missed anything?
>>>
>>> To all those who didn't join, if you disagree to anything in
>>> the Summary please voice your opinion.
>>>
>>> Also please let me know if someone wants to include an item in Next call's
>>> Agenda.
>>>
>>> Including the Summary here too (might potentially break formatting):
>>>
>>> *Key Decisions*
>>>
>>> - *API*
>>> - Progress:
>>> - Project Board: https://github.com/apache/airflow/projects/1
>>> - The issues labelled with "Enhancement" are not a
>>> requirement for 2.0
>>> - Endpoints:
>>> - Task Instance Endpoint
>>> <https://github.com/apache/airflow/pull/9597> is WIP, all
>>> the other endpoints have been implemented.
>>> - Permissions Model:
>>> - PR <https://github.com/apache/airflow/pull/10594> has
>>> been merged.
>>> - The next piece of work to be done is migrating existing
>>> Views to use resource-based permissions. (Github issue
>>> <https://github.com/apache/airflow/issues/10469>). This is
>>> mainly for standardizing the permissions model across API and UI.
>>> - *Providers*
>>> - Vote on AIP-8 took place on the mailing list
>>> <https://lists.apache.org/thread.html/rcd63bbe62a618c4547bd00b1c1d14dc329cfe1c09e4795571be28cb3%40%3Cdev.airflow.apache.org%3E>
>>> .
>>> - There is an ongoing discussion on the same thread about SemVer
>>> vs CalVer for the Providers package.
>>> - The people involved on the call were *leaning towards SemVer* to
>>> make a clear distinction about a breaking release. This will potentially
>>> increase the work on release managers but some automation around releasing
>>> (similar to backport providers) and automation around the generation of the
>>> changelog for the providers would make the effort less painful.
>>> - *Version Per Provide: *Each Providers package would have a
>>> separate versioning i.e. we might release "google-providers 3.1" and
>>> "amazon-providers" 3.7 at the same time but the versioning for a particular
>>> provider will be independent of other providers.
>>> - *DEV*
>>> - Would be good to have a release policy on when we can deprecate a
>>> feature, our release cadence. A good example is
>>> https://docs.djangoproject.com/en/3.1/internals/release-process/#release-cadence
>>> - *SubDag Deprecation*
>>> - There is a mailing list thread
>>> <https://lists.apache.org/thread.html/ra52746f9c8274469d343b5f0251199de776e75ab75ded6830886fb6a%40%3Cdev.airflow.apache.org%3E> on
>>> whether or not we want to deprecate SubDags in favor of Taskgroups, the
>>> majority on the call agreed that we *should not deprecate the
>>> Subdags yet* and wait till people have used TaskGroups and it has
>>> feature parity with SubDags.
>>> - However, we should *clearly recommend using TaskGroups compared
>>> to SubDags* in our docs and state limitations of the SubDags.
>>> - *Helm Chart Release*
>>> - Deferred until 2.0 is out
>>> - Will be available to use from the source code of Airflow on
>>> Github but the first official release of the Helm chart will only happen
>>> after Airflow 2.0
>>> - *Docs*
>>> - Mailing list thread
>>> <https://lists.apache.org/thread.html/rc6331d0bf97d91dc88853c992513f4e886f113c1cff030876996859e%40%3Cdev.airflow.apache.org%3E> to
>>> get some feedback has been created and cross-posted across Slack and
>>> Twitter. Once we have enough feedback, Kaxil will create Github issues for
>>> them so that anyone willing to help on it can start working on it.
>>> - A separate section for Upgrading to 2.0 would be ideal, can be
>>> a duplicate of Updating.md but with a better structure and more organized.
>>> - *UI Changes*
>>> - *Github Issue: *https://github.com/apache/airflow/issues/10953
>>> - There are some proposals from Ryan for the UI changes for which
>>> he has created some PRs (links below) and in the process of creating few
>>> more.
>>> - Task Instance Modal UX Enhancements · Issue #10944 ·
>>> apache/airflow <https://github.com/apache/airflow/pull/10944>
>>> - Replace JS package toggle w/ pure CSS solution #11035
>>> <https://github.com/apache/airflow/pull/11035>
>>> - Task Instance header/navigation pattern UX cleanup
>>> <https://github.com/apache/airflow/pull/11089> – Suggestions /
>>> VOTE needed here if anyone has strong opinions
>>> - *Scheduler HA*
>>> - *Reminder*: A draft PR for Scheduler HA
>>> <https://github.com/apache/airflow/pull/10956> is available for
>>> review. It would be good to get some more feedback from the wider community
>>> with their own DEV setup if possible.
>>> - *Process*
>>> - Any new PRs would continue to be merged until we complete the
>>> items for 2.0 and release alphas.
>>> - *NOTE: *The Timeline shown on the Planning page
>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning> will
>>> be revisited every week on the Dev Call and updated if needed based on the
>>> progress towards the major features of Airflow 2.0
>>>
>>>
>>> Regards,
>>> Kaxil
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #5 - 21 Sep 2020
Posted by Kaxil Naik <ka...@gmail.com>.
Hi Jarek,
Thanks agree with MySQL 8 part, I will try to tackle that in a couple of
days and help out with the other PRs / issues too. The Selective
Optimisation one is a very interesting one too and will take a look at the
WIP PR you have already created.
Regards,
Kaxil
On Sat, Sep 26, 2020 at 10:13 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Thanks, Kaxil, Good summary!
>
> Just a comment on progress here from my side. I have just updated the
> status of three issues that are relevant for our Monday discussion.
>
> * Enable MySQL 8 CI jobs #11164
> https://github.com/apache/airflow/issues/11164
>
> This one is a prerequisite IMHO to get HA Scheduler merged because we have
> a dependency on MySQL 8 for it but we do not test MySQL 8 on CI at all
> currently. I think this one should be implemented as part of the HA
> Scheduler PR merge preparation. I'd love to take that on, but I have no
> time to try it out even, but likely it is only adding the version and it
> should work, however, there is quite a risk that we will need some fixes at
> least in the tests and we for sure need to adapt Dockerfile to use MySQL 8
> client.
>
> It is also (not very strongly) depends on those three that are
> closely related. It would be best if those three are all completed because
> this will give us a chance to test the separation of packages and in case
> we are going Semver (which we all agree is a better approach), there is
> quite some work to implement SEMVER versioning for separate packages. This
> separation is also needed to massively speed up our CI builds and it will
> help us to tackle increased CI pressure when we add MySQL 8.
>
> * [OPTIMISATION] Selective builds for different types of tests #10507
> https://github.com/apache/airflow/issues/10507
> * Fully separate provider packages from the Airflow core (AIP-8)
> https://github.com/apache/airflow/issues/11163
> * Release a 2nd wave of Backport packages #10014
> https://github.com/apache/airflow/issues/10014
>
> Unfortunately, due to other obligations, I likely won't be able to
> complete any of those this week and I will have very little time this week
> in general, so any help on those is appreciated - if anyone would like to
> take any of those - I linked all the WIP PRs for those - so if there is
> anyone who would like to take it over - feel free.
>
> I will do my best to be able to take part in the meeting on Monday.
>
> J.
>
>
> On Fri, Sep 25, 2020 at 9:09 PM Kaxil Naik <ka...@apache.org> wrote:
>
>> Hi all,
>>
>> I have created a document to summarize the discussion from our dev call for
>> Airflow 2.0 this Monday. Apologies for the delay in publishing the Meeting
>> Notes.
>>
>> Thank you all who joined the call.
>>
>> *Doc Link*:
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#5:21Sep2020
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%235:21Sep2020>
>>
>> To all those who attended, can you please double-check and add if I have
>> missed anything?
>>
>> To all those who didn't join, if you disagree to anything in
>> the Summary please voice your opinion.
>>
>> Also please let me know if someone wants to include an item in Next call's
>> Agenda.
>>
>> Including the Summary here too (might potentially break formatting):
>>
>> *Key Decisions*
>>
>> - *API*
>> - Progress:
>> - Project Board: https://github.com/apache/airflow/projects/1
>> - The issues labelled with "Enhancement" are not a
>> requirement for 2.0
>> - Endpoints:
>> - Task Instance Endpoint
>> <https://github.com/apache/airflow/pull/9597> is WIP, all
>> the other endpoints have been implemented.
>> - Permissions Model:
>> - PR <https://github.com/apache/airflow/pull/10594> has been
>> merged.
>> - The next piece of work to be done is migrating existing
>> Views to use resource-based permissions. (Github issue
>> <https://github.com/apache/airflow/issues/10469>). This is
>> mainly for standardizing the permissions model across API and UI.
>> - *Providers*
>> - Vote on AIP-8 took place on the mailing list
>> <https://lists.apache.org/thread.html/rcd63bbe62a618c4547bd00b1c1d14dc329cfe1c09e4795571be28cb3%40%3Cdev.airflow.apache.org%3E>
>> .
>> - There is an ongoing discussion on the same thread about SemVer
>> vs CalVer for the Providers package.
>> - The people involved on the call were *leaning towards SemVer* to
>> make a clear distinction about a breaking release. This will potentially
>> increase the work on release managers but some automation around releasing
>> (similar to backport providers) and automation around the generation of the
>> changelog for the providers would make the effort less painful.
>> - *Version Per Provide: *Each Providers package would have a
>> separate versioning i.e. we might release "google-providers 3.1" and
>> "amazon-providers" 3.7 at the same time but the versioning for a particular
>> provider will be independent of other providers.
>> - *DEV*
>> - Would be good to have a release policy on when we can deprecate a
>> feature, our release cadence. A good example is
>> https://docs.djangoproject.com/en/3.1/internals/release-process/#release-cadence
>> - *SubDag Deprecation*
>> - There is a mailing list thread
>> <https://lists.apache.org/thread.html/ra52746f9c8274469d343b5f0251199de776e75ab75ded6830886fb6a%40%3Cdev.airflow.apache.org%3E> on
>> whether or not we want to deprecate SubDags in favor of Taskgroups, the
>> majority on the call agreed that we *should not deprecate the
>> Subdags yet* and wait till people have used TaskGroups and it has
>> feature parity with SubDags.
>> - However, we should *clearly recommend using TaskGroups compared
>> to SubDags* in our docs and state limitations of the SubDags.
>> - *Helm Chart Release*
>> - Deferred until 2.0 is out
>> - Will be available to use from the source code of Airflow on
>> Github but the first official release of the Helm chart will only happen
>> after Airflow 2.0
>> - *Docs*
>> - Mailing list thread
>> <https://lists.apache.org/thread.html/rc6331d0bf97d91dc88853c992513f4e886f113c1cff030876996859e%40%3Cdev.airflow.apache.org%3E> to
>> get some feedback has been created and cross-posted across Slack and
>> Twitter. Once we have enough feedback, Kaxil will create Github issues for
>> them so that anyone willing to help on it can start working on it.
>> - A separate section for Upgrading to 2.0 would be ideal, can be a
>> duplicate of Updating.md but with a better structure and more organized.
>> - *UI Changes*
>> - *Github Issue: *https://github.com/apache/airflow/issues/10953
>> - There are some proposals from Ryan for the UI changes for which
>> he has created some PRs (links below) and in the process of creating few
>> more.
>> - Task Instance Modal UX Enhancements · Issue #10944 ·
>> apache/airflow <https://github.com/apache/airflow/pull/10944>
>> - Replace JS package toggle w/ pure CSS solution #11035
>> <https://github.com/apache/airflow/pull/11035>
>> - Task Instance header/navigation pattern UX cleanup
>> <https://github.com/apache/airflow/pull/11089> – Suggestions /
>> VOTE needed here if anyone has strong opinions
>> - *Scheduler HA*
>> - *Reminder*: A draft PR for Scheduler HA
>> <https://github.com/apache/airflow/pull/10956> is available for
>> review. It would be good to get some more feedback from the wider community
>> with their own DEV setup if possible.
>> - *Process*
>> - Any new PRs would continue to be merged until we complete the
>> items for 2.0 and release alphas.
>> - *NOTE: *The Timeline shown on the Planning page
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning> will
>> be revisited every week on the Dev Call and updated if needed based on the
>> progress towards the major features of Airflow 2.0
>>
>>
>> Regards,
>> Kaxil
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: [Meeting Notes] Airflow 2.0 Dev Call #5 - 21 Sep 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
Thanks, Kaxil, Good summary!
Just a comment on progress here from my side. I have just updated the
status of three issues that are relevant for our Monday discussion.
* Enable MySQL 8 CI jobs #11164
https://github.com/apache/airflow/issues/11164
This one is a prerequisite IMHO to get HA Scheduler merged because we have
a dependency on MySQL 8 for it but we do not test MySQL 8 on CI at all
currently. I think this one should be implemented as part of the HA
Scheduler PR merge preparation. I'd love to take that on, but I have no
time to try it out even, but likely it is only adding the version and it
should work, however, there is quite a risk that we will need some fixes at
least in the tests and we for sure need to adapt Dockerfile to use MySQL 8
client.
It is also (not very strongly) depends on those three that are
closely related. It would be best if those three are all completed because
this will give us a chance to test the separation of packages and in case
we are going Semver (which we all agree is a better approach), there is
quite some work to implement SEMVER versioning for separate packages. This
separation is also needed to massively speed up our CI builds and it will
help us to tackle increased CI pressure when we add MySQL 8.
* [OPTIMISATION] Selective builds for different types of tests #10507
https://github.com/apache/airflow/issues/10507
* Fully separate provider packages from the Airflow core (AIP-8)
https://github.com/apache/airflow/issues/11163
* Release a 2nd wave of Backport packages #10014
https://github.com/apache/airflow/issues/10014
Unfortunately, due to other obligations, I likely won't be able to complete
any of those this week and I will have very little time this week in
general, so any help on those is appreciated - if anyone would like to take
any of those - I linked all the WIP PRs for those - so if there is anyone
who would like to take it over - feel free.
I will do my best to be able to take part in the meeting on Monday.
J.
On Fri, Sep 25, 2020 at 9:09 PM Kaxil Naik <ka...@apache.org> wrote:
> Hi all,
>
> I have created a document to summarize the discussion from our dev call for
> Airflow 2.0 this Monday. Apologies for the delay in publishing the Meeting
> Notes.
>
> Thank you all who joined the call.
>
> *Doc Link*:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#5:21Sep2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%235:21Sep2020>
>
> To all those who attended, can you please double-check and add if I have
> missed anything?
>
> To all those who didn't join, if you disagree to anything in
> the Summary please voice your opinion.
>
> Also please let me know if someone wants to include an item in Next call's
> Agenda.
>
> Including the Summary here too (might potentially break formatting):
>
> *Key Decisions*
>
> - *API*
> - Progress:
> - Project Board: https://github.com/apache/airflow/projects/1
> - The issues labelled with "Enhancement" are not a
> requirement for 2.0
> - Endpoints:
> - Task Instance Endpoint
> <https://github.com/apache/airflow/pull/9597> is WIP, all the
> other endpoints have been implemented.
> - Permissions Model:
> - PR <https://github.com/apache/airflow/pull/10594> has been
> merged.
> - The next piece of work to be done is migrating existing
> Views to use resource-based permissions. (Github issue
> <https://github.com/apache/airflow/issues/10469>). This is
> mainly for standardizing the permissions model across API and UI.
> - *Providers*
> - Vote on AIP-8 took place on the mailing list
> <https://lists.apache.org/thread.html/rcd63bbe62a618c4547bd00b1c1d14dc329cfe1c09e4795571be28cb3%40%3Cdev.airflow.apache.org%3E>
> .
> - There is an ongoing discussion on the same thread about SemVer vs
> CalVer for the Providers package.
> - The people involved on the call were *leaning towards SemVer* to
> make a clear distinction about a breaking release. This will potentially
> increase the work on release managers but some automation around releasing
> (similar to backport providers) and automation around the generation of the
> changelog for the providers would make the effort less painful.
> - *Version Per Provide: *Each Providers package would have a
> separate versioning i.e. we might release "google-providers 3.1" and
> "amazon-providers" 3.7 at the same time but the versioning for a particular
> provider will be independent of other providers.
> - *DEV*
> - Would be good to have a release policy on when we can deprecate a
> feature, our release cadence. A good example is
> https://docs.djangoproject.com/en/3.1/internals/release-process/#release-cadence
> - *SubDag Deprecation*
> - There is a mailing list thread
> <https://lists.apache.org/thread.html/ra52746f9c8274469d343b5f0251199de776e75ab75ded6830886fb6a%40%3Cdev.airflow.apache.org%3E> on
> whether or not we want to deprecate SubDags in favor of Taskgroups, the
> majority on the call agreed that we *should not deprecate the
> Subdags yet* and wait till people have used TaskGroups and it has
> feature parity with SubDags.
> - However, we should *clearly recommend using TaskGroups compared
> to SubDags* in our docs and state limitations of the SubDags.
> - *Helm Chart Release*
> - Deferred until 2.0 is out
> - Will be available to use from the source code of Airflow on
> Github but the first official release of the Helm chart will only happen
> after Airflow 2.0
> - *Docs*
> - Mailing list thread
> <https://lists.apache.org/thread.html/rc6331d0bf97d91dc88853c992513f4e886f113c1cff030876996859e%40%3Cdev.airflow.apache.org%3E> to
> get some feedback has been created and cross-posted across Slack and
> Twitter. Once we have enough feedback, Kaxil will create Github issues for
> them so that anyone willing to help on it can start working on it.
> - A separate section for Upgrading to 2.0 would be ideal, can be a
> duplicate of Updating.md but with a better structure and more organized.
> - *UI Changes*
> - *Github Issue: *https://github.com/apache/airflow/issues/10953
> - There are some proposals from Ryan for the UI changes for which
> he has created some PRs (links below) and in the process of creating few
> more.
> - Task Instance Modal UX Enhancements · Issue #10944 ·
> apache/airflow <https://github.com/apache/airflow/pull/10944>
> - Replace JS package toggle w/ pure CSS solution #11035
> <https://github.com/apache/airflow/pull/11035>
> - Task Instance header/navigation pattern UX cleanup
> <https://github.com/apache/airflow/pull/11089> – Suggestions /
> VOTE needed here if anyone has strong opinions
> - *Scheduler HA*
> - *Reminder*: A draft PR for Scheduler HA
> <https://github.com/apache/airflow/pull/10956> is available for
> review. It would be good to get some more feedback from the wider community
> with their own DEV setup if possible.
> - *Process*
> - Any new PRs would continue to be merged until we complete the
> items for 2.0 and release alphas.
> - *NOTE: *The Timeline shown on the Planning page
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning> will
> be revisited every week on the Dev Call and updated if needed based on the
> progress towards the major features of Airflow 2.0
>
>
> Regards,
> Kaxil
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>