You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2020/09/02 07:16:26 UTC
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
As promised during the last call I prepared the proposal on how we can
approach the package model for Airflow 2.0 - including the "Provider
Packages" approach.
https://s.apache.org/airflow-2-0-package-model
I would like to discuss it at our next meeting on Monday. I'd love to
hear your comments.
J.
On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <Ja...@polidea.com> wrote:
>
> +1 Kevin on the call :).
>
> On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com> wrote:
>>
>> Thanks Kevin, Looking forward to see you on the next call.
>>
>> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
>>
>> > Thank you Kaxil, this is extremely helpful. We'll try to join at least the
>> > next meeting trying to see if we can provide more perspectives on
>> > SmartSensor and anything else we can help.
>> >
>> >
>> > Cheers,
>> > Kevin Y
>> >
>> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com> wrote:
>> >
>> > > Hi all,
>> > >
>> > > I have created a document to summarize the discussion from our second dev
>> > > call for Airflow 2.0.
>> > >
>> > > Thank you all who joined the call.
>> > >
>> > > *Doc Link*:
>> > >
>> > >
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
>> > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
>> > > <
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>> > >
>> > >
>> > > To all those who attended, can you please double-check and add if I have
>> > > missed anything?
>> > >
>> > > To all those who didn't join, if you disagree to anything in the Summary
>> > > please voice your opinion.
>> > >
>> > > Including the Summary here too (might potentially break formatting):
>> > >
>> > > *Key Decisions*
>> > >
>> > > - *Smart Sensors – *in 2.0 or 2.1
>> > > - AIP-17
>> > > <
>> > >
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
>> > > >
>> > > |
>> > > PR: https://github.com/apache/airflow/pull/5499
>> > > - We have not come to a conclusion yet on whether this should be
>> > > included in 2.0 or not. The majority is towards adding it in 2.0
>> > (as
>> > > it
>> > > supports Airflow 2.0's Scalability story) and marking it as
>> > > *experimental*.
>> > > - There were some questions raised around supporting this new
>> > > feature. So we decided that *everyone would take a look at the PR
>> > > itself and we will spend a few minutes in the next meeting to
>> > decide
>> > > whether it is 2.0 or not*.
>> > > - *Simplification of KubernetesExecutor / KubernetesPodOperator*
>> > > - PR: https://github.com/apache/airflow/pull/10393
>> > > - This will be part of *Airflow 2.0*
>> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
>> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 | Design
>> > > Doc:
>> > >
>> > >
>> > https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
>> > > - *Scope*:
>> > > - Users bash script won’t be included but anything in the core
>> > > Airflow would be covered
>> > > -
>> > >
>> > > *DAG Definitions*:
>> > > - Changes in Path for contrib to Providers packages
>> > > - DAG Interfaces: changes in arguments of a DAG /
>> > BaseOperator
>> > > - *Configurations*:
>> > > - Option to auto-replace deprecated configs with new options
>> > > - *Run-time Core items*:
>> > > - Changes like "Connection type can't be null". The
>> > > upgrade-check should at least shown warning if it can't
>> > > provide option to
>> > > detect the type.
>> > > - *CLI refactor is out-of-scope*
>> > > - Automatic refactor is *out-of-scope* as it is too difficult
>> > > to cover all the cases in the Users bash scripts.
>> > > - This will be covered by docs or by showing warnings via the
>> > > upgrade-check command
>> > > - *Experimental API to New API refactor is out-of-scope* (will
>> > be
>> > > covered by Migration docs)
>> > > - We agreed that the airflow upgrade-check command *needs to be
>> > > available in the last release before Airflow 2.0* (1.10.x or
>> > 1.11.x)
>> > > - Potential problems with time-consuming DB Migration were also
>> > > discussed. If we identify such a DB Migration (example the one
>> > involving
>> > > TaskInstance table) should be noted separately in Updating.md to
>> > > provide a
>> > > warning to the users.
>> > > - *DEV Calls Feedback*
>> > > - We agreed on having *Weekly calls from 7 September onwards*
>> > > - Calls will start with a 5-min reviewing the progress from the
>> > last
>> > > call towards 2.0
>> > > - *Process*
>> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
>> > > - Changelog:
>> > > - The current way of Changelog is OK. We don't need further
>> > > categorization like Webserver, Scheduler etc.
>> > > - Separate Changelog would be created for Providers Packages
>> > > - We need to figure a way to tag/label PRs & Issues with correct
>> > > categories. Some options that were discussed were:
>> > > - Adding labels on the PRs & Issues via Bot
>> > > - A field in PR template for PR authors to add, the bot would
>> > > then read the field which would be used to label the PR
>> > > - Add rules, for example Committers needs to add appropriate
>> > > labels to the PR before merging it. We could have a
>> > > scheduled Github
>> > > Actions workflow that would fail if it finds PRs without
>> > > labels.
>> > >
>> > > *Things to Discuss Next*
>> > >
>> > > - *7 September*
>> > > - Progress, Current Work & Discussions
>> > > - API
>> > > - Providers Packages
>> > > - Discuss open questions
>> > > - Improvements to SubDags / Concept of TaskGroup
>> > > - AIP-34 <https://github.com/apache/airflow/pull/10153>
>> > > - *14 September*
>> > > - Process:
>> > > - When should we defer the in-scope items to post-2.0
>> > > - Completion by a date?
>> > > - Progress by a date?
>> > > - Progress, Current Work & Discussions
>> > > - Scheduler HA
>> > > - Docs Improvements
>> > > - Helm Chart
>> > > - Discuss the issue with sources
>> > >
>> > >
>> > > Regards,
>> > > Kaxil
>> > >
>> >
>
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>
>
--
Jarek Potiuk
Polidea | Principal Software Engineer
M: +48 660 796 129
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
Yep. Thanks, Ash - indeed that is closely related and if do this, I think
9506 should be fixed together. I added it to the doc.
I also resolved some of the questions and comments in the doc and added a
short "Alternative" - i.e. what is the alternative - monolith Airflow
release as we did so far.
I think at the call today we should start from this "alternative" - because
maybe the gains that we have by releasing separate packages are not as big
as the complexity we are adding.
I am a bit on the fence on that one myself - I am not 100 % sure if the
gains are really as big, I would love to hear what others think about it :)
J.
On Sat, Sep 5, 2020 at 11:28 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> Very closely related issue https://github.com/apache/airflow/issues/9506
>
> On 5 September 2020 08:01:11 BST, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >And we have a new addition from Kamil about the need to extend slightly
> >plugin mechanism to be able to cover dynamically "Connections",
> >"Connection
> >Form" and "Extra Links" - those are indeed the "core -> Providers"
> >dependencies that we still have.
> >
> >They seem to be easy to handle by making providers "plugins" and
> >extending
> >the plugin mechanism a bit. Thanks Kamil for the thoughtful input!
> >
> >On Fri, Sep 4, 2020 at 8:08 PM Jarek Potiuk <Ja...@polidea.com>
> >wrote:
> >
> >> Just a short reminder - for some more comments/review on the "PIP
> >package
> >> model of Airflow 2.0" doc
> >>
> >>
> >
> https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit#
> >>
> >> I've added one small addition - in this model we want to make sure
> >that
> >> there are no dependencies of core packages on any of the providers -
> >we do
> >> not run such checks yet but it's easy to add.
> >>
> >> J
> >>
> >>
> >> On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk
> ><Ja...@polidea.com>
> >> wrote:
> >>
> >>> Cool!
> >>>
> >>> If you have comments on particular sections/paragraphs - it's easier
> >to
> >>> keep track of it and respond in the doc. If you have some general
> >>> statements, and some summary of your thinking after the review -
> >it's best
> >>> to respond to the email :)
> >>>
> >>> I am ok with both and will aggregate it eventually.
> >>>
> >>> J.
> >>>
> >>>
> >>> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io>
> >wrote:
> >>>
> >>>> Jarek,
> >>>>
> >>>> Thank you, this is very helpful.
> >>>> I assume that you would like comments in the document itself?
> >>>> Or, would you like them in email?
> >>>>
> >>>> Best regards,
> >>>> Vikram
> >>>>
> >>>>
> >>>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk
> ><Ja...@polidea.com>
> >>>> wrote:
> >>>>
> >>>> > As promised during the last call I prepared the proposal on how
> >we can
> >>>> > approach the package model for Airflow 2.0 - including the
> >"Provider
> >>>> > Packages" approach.
> >>>> >
> >>>> > https://s.apache.org/airflow-2-0-package-model
> >>>> >
> >>>> > I would like to discuss it at our next meeting on Monday. I'd
> >love to
> >>>> > hear your comments.
> >>>> >
> >>>> > J.
> >>>> >
> >>>> >
> >>>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <
> >>>> Jarek.Potiuk@polidea.com>
> >>>> > wrote:
> >>>> > >
> >>>> > > +1 Kevin on the call :).
> >>>> > >
> >>>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik
> ><ka...@gmail.com>
> >>>> wrote:
> >>>> > >>
> >>>> > >> Thanks Kevin, Looking forward to see you on the next call.
> >>>> > >>
> >>>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com>
> >wrote:
> >>>> > >>
> >>>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to
> >join at
> >>>> > least the
> >>>> > >> > next meeting trying to see if we can provide more
> >perspectives on
> >>>> > >> > SmartSensor and anything else we can help.
> >>>> > >> >
> >>>> > >> >
> >>>> > >> > Cheers,
> >>>> > >> > Kevin Y
> >>>> > >> >
> >>>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik
> ><ka...@gmail.com>
> >>>> > wrote:
> >>>> > >> >
> >>>> > >> > > Hi all,
> >>>> > >> > >
> >>>> > >> > > I have created a document to summarize the discussion from
> >our
> >>>> > second dev
> >>>> > >> > > call for Airflow 2.0.
> >>>> > >> > >
> >>>> > >> > > Thank you all who joined the call.
> >>>> > >> > >
> >>>> > >> > > *Doc Link*:
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> >
> >>>> >
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
> >>>>
> ><
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >
> >>>> > <
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >>>> >
> >>>> > >> > <
> >>>> >
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >>>> > >
> >>>> > >> > > <
> >>>> > >> >
> >>>> >
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > > To all those who attended, can you please double-check and
> >add
> >>>> if I
> >>>> > have
> >>>> > >> > > missed anything?
> >>>> > >> > >
> >>>> > >> > > To all those who didn't join, if you disagree to anything
> >in the
> >>>> > Summary
> >>>> > >> > > please voice your opinion.
> >>>> > >> > >
> >>>> > >> > > Including the Summary here too (might potentially break
> >>>> formatting):
> >>>> > >> > >
> >>>> > >> > > *Key Decisions*
> >>>> > >> > >
> >>>> > >> > > - *Smart Sensors – *in 2.0 or 2.1
> >>>> > >> > > - AIP-17
> >>>> > >> > > <
> >>>> > >> > >
> >>>> > >> >
> >>>> >
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
> >>>> > >> > > >
> >>>> > >> > > |
> >>>> > >> > > PR: https://github.com/apache/airflow/pull/5499
> >>>> > >> > > - We have not come to a conclusion yet on whether
> >this
> >>>> should
> >>>> > be
> >>>> > >> > > included in 2.0 or not. The majority is towards
> >adding it
> >>>> in
> >>>> > 2.0
> >>>> > >> > (as
> >>>> > >> > > it
> >>>> > >> > > supports Airflow 2.0's Scalability story) and
> >marking it
> >>>> as
> >>>> > >> > > *experimental*.
> >>>> > >> > > - There were some questions raised around supporting
> >this
> >>>> new
> >>>> > >> > > feature. So we decided that *everyone would take a
> >look at
> >>>> > the PR
> >>>> > >> > > itself and we will spend a few minutes in the next
> >>>> meeting to
> >>>> > >> > decide
> >>>> > >> > > whether it is 2.0 or not*.
> >>>> > >> > > - *Simplification of KubernetesExecutor /
> >>>> KubernetesPodOperator*
> >>>> > >> > > - PR: https://github.com/apache/airflow/pull/10393
> >>>> > >> > > - This will be part of *Airflow 2.0*
> >>>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)*
> >command *
> >>>> > >> > > - WIP PR: PR:
> >https://github.com/apache/airflow/pull/9467
> >>>> |
> >>>> > Design
> >>>> > >> > > Doc:
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> >
> >>>> >
> >>>>
> >
> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
> >>>> > >> > > - *Scope*:
> >>>> > >> > > - Users bash script won’t be included but
> >anything in
> >>>> the
> >>>> > core
> >>>> > >> > > Airflow would be covered
> >>>> > >> > > -
> >>>> > >> > >
> >>>> > >> > > *DAG Definitions*:
> >>>> > >> > > - Changes in Path for contrib to Providers
> >packages
> >>>> > >> > > - DAG Interfaces: changes in arguments of a
> >DAG /
> >>>> > >> > BaseOperator
> >>>> > >> > > - *Configurations*:
> >>>> > >> > > - Option to auto-replace deprecated configs
> >with new
> >>>> > options
> >>>> > >> > > - *Run-time Core items*:
> >>>> > >> > > - Changes like "Connection type can't be
> >null". The
> >>>> > >> > > upgrade-check should at least shown warning if
> >it
> >>>> can't
> >>>> > >> > > provide option to
> >>>> > >> > > detect the type.
> >>>> > >> > > - *CLI refactor is out-of-scope*
> >>>> > >> > > - Automatic refactor is *out-of-scope* as it
> >is too
> >>>> > difficult
> >>>> > >> > > to cover all the cases in the Users bash
> >scripts.
> >>>> > >> > > - This will be covered by docs or by showing
> >>>> warnings
> >>>> > via the
> >>>> > >> > > upgrade-check command
> >>>> > >> > > - *Experimental API to New API refactor is
> >>>> out-of-scope*
> >>>> > (will
> >>>> > >> > be
> >>>> > >> > > covered by Migration docs)
> >>>> > >> > > - We agreed that the airflow upgrade-check command
> >*needs
> >>>> to
> >>>> > be
> >>>> > >> > > available in the last release before Airflow 2.0*
> >(1.10.x
> >>>> or
> >>>> > >> > 1.11.x)
> >>>> > >> > > - Potential problems with time-consuming DB Migration
> >were
> >>>> also
> >>>> > >> > > discussed. If we identify such a DB Migration (example
> >the
> >>>> one
> >>>> > >> > involving
> >>>> > >> > > TaskInstance table) should be noted separately in
> >>>> Updating.md to
> >>>> > >> > > provide a
> >>>> > >> > > warning to the users.
> >>>> > >> > > - *DEV Calls Feedback*
> >>>> > >> > > - We agreed on having *Weekly calls from 7 September
> >>>> onwards*
> >>>> > >> > > - Calls will start with a 5-min reviewing the
> >progress
> >>>> from
> >>>> > the
> >>>> > >> > last
> >>>> > >> > > call towards 2.0
> >>>> > >> > > - *Process*
> >>>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep
> >2020
> >>>> > >> > > - Changelog:
> >>>> > >> > > - The current way of Changelog is OK. We don't
> >need
> >>>> further
> >>>> > >> > > categorization like Webserver, Scheduler etc.
> >>>> > >> > > - Separate Changelog would be created for
> >Providers
> >>>> > Packages
> >>>> > >> > > - We need to figure a way to tag/label PRs &
> >Issues
> >>>> with
> >>>> > correct
> >>>> > >> > > categories. Some options that were discussed
> >were:
> >>>> > >> > > - Adding labels on the PRs & Issues via Bot
> >>>> > >> > > - A field in PR template for PR authors to
> >add, the
> >>>> bot
> >>>> > would
> >>>> > >> > > then read the field which would be used to
> >label
> >>>> the PR
> >>>> > >> > > - Add rules, for example Committers needs to
> >add
> >>>> > appropriate
> >>>> > >> > > labels to the PR before merging it. We could
> >have a
> >>>> > >> > > scheduled Github
> >>>> > >> > > Actions workflow that would fail if it finds
> >PRs
> >>>> without
> >>>> > >> > > labels.
> >>>> > >> > >
> >>>> > >> > > *Things to Discuss Next*
> >>>> > >> > >
> >>>> > >> > > - *7 September*
> >>>> > >> > > - Progress, Current Work & Discussions
> >>>> > >> > > - API
> >>>> > >> > > - Providers Packages
> >>>> > >> > > - Discuss open questions
> >>>> > >> > > - Improvements to SubDags / Concept of TaskGroup
> >>>> > >> > > - AIP-34 <
> >>>> https://github.com/apache/airflow/pull/10153>
> >>>> > >> > > - *14 September*
> >>>> > >> > > - Process:
> >>>> > >> > > - When should we defer the in-scope items to
> >post-2.0
> >>>> > >> > > - Completion by a date?
> >>>> > >> > > - Progress by a date?
> >>>> > >> > > - Progress, Current Work & Discussions
> >>>> > >> > > - Scheduler HA
> >>>> > >> > > - Docs Improvements
> >>>> > >> > > - Helm Chart
> >>>> > >> > > - Discuss the issue with sources
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > > Regards,
> >>>> > >> > > Kaxil
> >>>> > >> > >
> >>>> > >> >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > --
> >>>> > >
> >>>> > > Jarek Potiuk
> >>>> > > Polidea | Principal Software Engineer
> >>>> > >
> >>>> > > M: +48 660 796 129
> >>>> > >
> >>>> > >
> >>>> >
> >>>> >
> >>>> > --
> >>>> >
> >>>> > Jarek Potiuk
> >>>> > Polidea | Principal Software Engineer
> >>>> >
> >>>> > M: +48 660 796 129
> >>>> >
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>> Jarek Potiuk
> >>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>
> >>> M: +48 660 796 129 <+48660796129>
> >>> [image: Polidea] <https://www.polidea.com/>
> >>>
> >>>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >>
> >
> >--
> >
> >Jarek Potiuk
> >Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> >M: +48 660 796 129 <+48660796129>
> >[image: Polidea] <https://www.polidea.com/>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Ash Berlin-Taylor <as...@apache.org>.
Very closely related issue https://github.com/apache/airflow/issues/9506
On 5 September 2020 08:01:11 BST, Jarek Potiuk <Ja...@polidea.com> wrote:
>And we have a new addition from Kamil about the need to extend slightly
>plugin mechanism to be able to cover dynamically "Connections",
>"Connection
>Form" and "Extra Links" - those are indeed the "core -> Providers"
>dependencies that we still have.
>
>They seem to be easy to handle by making providers "plugins" and
>extending
>the plugin mechanism a bit. Thanks Kamil for the thoughtful input!
>
>On Fri, Sep 4, 2020 at 8:08 PM Jarek Potiuk <Ja...@polidea.com>
>wrote:
>
>> Just a short reminder - for some more comments/review on the "PIP
>package
>> model of Airflow 2.0" doc
>>
>>
>https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit#
>>
>> I've added one small addition - in this model we want to make sure
>that
>> there are no dependencies of core packages on any of the providers -
>we do
>> not run such checks yet but it's easy to add.
>>
>> J
>>
>>
>> On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk
><Ja...@polidea.com>
>> wrote:
>>
>>> Cool!
>>>
>>> If you have comments on particular sections/paragraphs - it's easier
>to
>>> keep track of it and respond in the doc. If you have some general
>>> statements, and some summary of your thinking after the review -
>it's best
>>> to respond to the email :)
>>>
>>> I am ok with both and will aggregate it eventually.
>>>
>>> J.
>>>
>>>
>>> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io>
>wrote:
>>>
>>>> Jarek,
>>>>
>>>> Thank you, this is very helpful.
>>>> I assume that you would like comments in the document itself?
>>>> Or, would you like them in email?
>>>>
>>>> Best regards,
>>>> Vikram
>>>>
>>>>
>>>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk
><Ja...@polidea.com>
>>>> wrote:
>>>>
>>>> > As promised during the last call I prepared the proposal on how
>we can
>>>> > approach the package model for Airflow 2.0 - including the
>"Provider
>>>> > Packages" approach.
>>>> >
>>>> > https://s.apache.org/airflow-2-0-package-model
>>>> >
>>>> > I would like to discuss it at our next meeting on Monday. I'd
>love to
>>>> > hear your comments.
>>>> >
>>>> > J.
>>>> >
>>>> >
>>>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <
>>>> Jarek.Potiuk@polidea.com>
>>>> > wrote:
>>>> > >
>>>> > > +1 Kevin on the call :).
>>>> > >
>>>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik
><ka...@gmail.com>
>>>> wrote:
>>>> > >>
>>>> > >> Thanks Kevin, Looking forward to see you on the next call.
>>>> > >>
>>>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com>
>wrote:
>>>> > >>
>>>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to
>join at
>>>> > least the
>>>> > >> > next meeting trying to see if we can provide more
>perspectives on
>>>> > >> > SmartSensor and anything else we can help.
>>>> > >> >
>>>> > >> >
>>>> > >> > Cheers,
>>>> > >> > Kevin Y
>>>> > >> >
>>>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik
><ka...@gmail.com>
>>>> > wrote:
>>>> > >> >
>>>> > >> > > Hi all,
>>>> > >> > >
>>>> > >> > > I have created a document to summarize the discussion from
>our
>>>> > second dev
>>>> > >> > > call for Airflow 2.0.
>>>> > >> > >
>>>> > >> > > Thank you all who joined the call.
>>>> > >> > >
>>>> > >> > > *Doc Link*:
>>>> > >> > >
>>>> > >> > >
>>>> > >> >
>>>> >
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
>>>>
><https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
>>>> > <
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>>> >
>>>> > >> > <
>>>> >
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>>> > >
>>>> > >> > > <
>>>> > >> >
>>>> >
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>>> > >> > >
>>>> > >> > >
>>>> > >> > > To all those who attended, can you please double-check and
>add
>>>> if I
>>>> > have
>>>> > >> > > missed anything?
>>>> > >> > >
>>>> > >> > > To all those who didn't join, if you disagree to anything
>in the
>>>> > Summary
>>>> > >> > > please voice your opinion.
>>>> > >> > >
>>>> > >> > > Including the Summary here too (might potentially break
>>>> formatting):
>>>> > >> > >
>>>> > >> > > *Key Decisions*
>>>> > >> > >
>>>> > >> > > - *Smart Sensors – *in 2.0 or 2.1
>>>> > >> > > - AIP-17
>>>> > >> > > <
>>>> > >> > >
>>>> > >> >
>>>> >
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
>>>> > >> > > >
>>>> > >> > > |
>>>> > >> > > PR: https://github.com/apache/airflow/pull/5499
>>>> > >> > > - We have not come to a conclusion yet on whether
>this
>>>> should
>>>> > be
>>>> > >> > > included in 2.0 or not. The majority is towards
>adding it
>>>> in
>>>> > 2.0
>>>> > >> > (as
>>>> > >> > > it
>>>> > >> > > supports Airflow 2.0's Scalability story) and
>marking it
>>>> as
>>>> > >> > > *experimental*.
>>>> > >> > > - There were some questions raised around supporting
>this
>>>> new
>>>> > >> > > feature. So we decided that *everyone would take a
>look at
>>>> > the PR
>>>> > >> > > itself and we will spend a few minutes in the next
>>>> meeting to
>>>> > >> > decide
>>>> > >> > > whether it is 2.0 or not*.
>>>> > >> > > - *Simplification of KubernetesExecutor /
>>>> KubernetesPodOperator*
>>>> > >> > > - PR: https://github.com/apache/airflow/pull/10393
>>>> > >> > > - This will be part of *Airflow 2.0*
>>>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)*
>command *
>>>> > >> > > - WIP PR: PR:
>https://github.com/apache/airflow/pull/9467
>>>> |
>>>> > Design
>>>> > >> > > Doc:
>>>> > >> > >
>>>> > >> > >
>>>> > >> >
>>>> >
>>>>
>https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
>>>> > >> > > - *Scope*:
>>>> > >> > > - Users bash script won’t be included but
>anything in
>>>> the
>>>> > core
>>>> > >> > > Airflow would be covered
>>>> > >> > > -
>>>> > >> > >
>>>> > >> > > *DAG Definitions*:
>>>> > >> > > - Changes in Path for contrib to Providers
>packages
>>>> > >> > > - DAG Interfaces: changes in arguments of a
>DAG /
>>>> > >> > BaseOperator
>>>> > >> > > - *Configurations*:
>>>> > >> > > - Option to auto-replace deprecated configs
>with new
>>>> > options
>>>> > >> > > - *Run-time Core items*:
>>>> > >> > > - Changes like "Connection type can't be
>null". The
>>>> > >> > > upgrade-check should at least shown warning if
>it
>>>> can't
>>>> > >> > > provide option to
>>>> > >> > > detect the type.
>>>> > >> > > - *CLI refactor is out-of-scope*
>>>> > >> > > - Automatic refactor is *out-of-scope* as it
>is too
>>>> > difficult
>>>> > >> > > to cover all the cases in the Users bash
>scripts.
>>>> > >> > > - This will be covered by docs or by showing
>>>> warnings
>>>> > via the
>>>> > >> > > upgrade-check command
>>>> > >> > > - *Experimental API to New API refactor is
>>>> out-of-scope*
>>>> > (will
>>>> > >> > be
>>>> > >> > > covered by Migration docs)
>>>> > >> > > - We agreed that the airflow upgrade-check command
>*needs
>>>> to
>>>> > be
>>>> > >> > > available in the last release before Airflow 2.0*
>(1.10.x
>>>> or
>>>> > >> > 1.11.x)
>>>> > >> > > - Potential problems with time-consuming DB Migration
>were
>>>> also
>>>> > >> > > discussed. If we identify such a DB Migration (example
>the
>>>> one
>>>> > >> > involving
>>>> > >> > > TaskInstance table) should be noted separately in
>>>> Updating.md to
>>>> > >> > > provide a
>>>> > >> > > warning to the users.
>>>> > >> > > - *DEV Calls Feedback*
>>>> > >> > > - We agreed on having *Weekly calls from 7 September
>>>> onwards*
>>>> > >> > > - Calls will start with a 5-min reviewing the
>progress
>>>> from
>>>> > the
>>>> > >> > last
>>>> > >> > > call towards 2.0
>>>> > >> > > - *Process*
>>>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep
>2020
>>>> > >> > > - Changelog:
>>>> > >> > > - The current way of Changelog is OK. We don't
>need
>>>> further
>>>> > >> > > categorization like Webserver, Scheduler etc.
>>>> > >> > > - Separate Changelog would be created for
>Providers
>>>> > Packages
>>>> > >> > > - We need to figure a way to tag/label PRs &
>Issues
>>>> with
>>>> > correct
>>>> > >> > > categories. Some options that were discussed
>were:
>>>> > >> > > - Adding labels on the PRs & Issues via Bot
>>>> > >> > > - A field in PR template for PR authors to
>add, the
>>>> bot
>>>> > would
>>>> > >> > > then read the field which would be used to
>label
>>>> the PR
>>>> > >> > > - Add rules, for example Committers needs to
>add
>>>> > appropriate
>>>> > >> > > labels to the PR before merging it. We could
>have a
>>>> > >> > > scheduled Github
>>>> > >> > > Actions workflow that would fail if it finds
>PRs
>>>> without
>>>> > >> > > labels.
>>>> > >> > >
>>>> > >> > > *Things to Discuss Next*
>>>> > >> > >
>>>> > >> > > - *7 September*
>>>> > >> > > - Progress, Current Work & Discussions
>>>> > >> > > - API
>>>> > >> > > - Providers Packages
>>>> > >> > > - Discuss open questions
>>>> > >> > > - Improvements to SubDags / Concept of TaskGroup
>>>> > >> > > - AIP-34 <
>>>> https://github.com/apache/airflow/pull/10153>
>>>> > >> > > - *14 September*
>>>> > >> > > - Process:
>>>> > >> > > - When should we defer the in-scope items to
>post-2.0
>>>> > >> > > - Completion by a date?
>>>> > >> > > - Progress by a date?
>>>> > >> > > - Progress, Current Work & Discussions
>>>> > >> > > - Scheduler HA
>>>> > >> > > - Docs Improvements
>>>> > >> > > - Helm Chart
>>>> > >> > > - Discuss the issue with sources
>>>> > >> > >
>>>> > >> > >
>>>> > >> > > Regards,
>>>> > >> > > Kaxil
>>>> > >> > >
>>>> > >> >
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > >
>>>> > > Jarek Potiuk
>>>> > > Polidea | Principal Software Engineer
>>>> > >
>>>> > > M: +48 660 796 129
>>>> > >
>>>> > >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > Jarek Potiuk
>>>> > Polidea | Principal Software Engineer
>>>> >
>>>> > M: +48 660 796 129
>>>> >
>>>>
>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
>--
>
>Jarek Potiuk
>Polidea <https://www.polidea.com/> | Principal Software Engineer
>
>M: +48 660 796 129 <+48660796129>
>[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
And we have a new addition from Kamil about the need to extend slightly
plugin mechanism to be able to cover dynamically "Connections", "Connection
Form" and "Extra Links" - those are indeed the "core -> Providers"
dependencies that we still have.
They seem to be easy to handle by making providers "plugins" and extending
the plugin mechanism a bit. Thanks Kamil for the thoughtful input!
On Fri, Sep 4, 2020 at 8:08 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Just a short reminder - for some more comments/review on the "PIP package
> model of Airflow 2.0" doc
>
> https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit#
>
> I've added one small addition - in this model we want to make sure that
> there are no dependencies of core packages on any of the providers - we do
> not run such checks yet but it's easy to add.
>
> J
>
>
> On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Cool!
>>
>> If you have comments on particular sections/paragraphs - it's easier to
>> keep track of it and respond in the doc. If you have some general
>> statements, and some summary of your thinking after the review - it's best
>> to respond to the email :)
>>
>> I am ok with both and will aggregate it eventually.
>>
>> J.
>>
>>
>> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io> wrote:
>>
>>> Jarek,
>>>
>>> Thank you, this is very helpful.
>>> I assume that you would like comments in the document itself?
>>> Or, would you like them in email?
>>>
>>> Best regards,
>>> Vikram
>>>
>>>
>>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>> > As promised during the last call I prepared the proposal on how we can
>>> > approach the package model for Airflow 2.0 - including the "Provider
>>> > Packages" approach.
>>> >
>>> > https://s.apache.org/airflow-2-0-package-model
>>> >
>>> > I would like to discuss it at our next meeting on Monday. I'd love to
>>> > hear your comments.
>>> >
>>> > J.
>>> >
>>> >
>>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com>
>>> > wrote:
>>> > >
>>> > > +1 Kevin on the call :).
>>> > >
>>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com>
>>> wrote:
>>> > >>
>>> > >> Thanks Kevin, Looking forward to see you on the next call.
>>> > >>
>>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
>>> > >>
>>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to join at
>>> > least the
>>> > >> > next meeting trying to see if we can provide more perspectives on
>>> > >> > SmartSensor and anything else we can help.
>>> > >> >
>>> > >> >
>>> > >> > Cheers,
>>> > >> > Kevin Y
>>> > >> >
>>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com>
>>> > wrote:
>>> > >> >
>>> > >> > > Hi all,
>>> > >> > >
>>> > >> > > I have created a document to summarize the discussion from our
>>> > second dev
>>> > >> > > call for Airflow 2.0.
>>> > >> > >
>>> > >> > > Thank you all who joined the call.
>>> > >> > >
>>> > >> > > *Doc Link*:
>>> > >> > >
>>> > >> > >
>>> > >> >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
>>> > <
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>> >
>>> > >> > <
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>> > >
>>> > >> > > <
>>> > >> >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>> > >> > >
>>> > >> > >
>>> > >> > > To all those who attended, can you please double-check and add
>>> if I
>>> > have
>>> > >> > > missed anything?
>>> > >> > >
>>> > >> > > To all those who didn't join, if you disagree to anything in the
>>> > Summary
>>> > >> > > please voice your opinion.
>>> > >> > >
>>> > >> > > Including the Summary here too (might potentially break
>>> formatting):
>>> > >> > >
>>> > >> > > *Key Decisions*
>>> > >> > >
>>> > >> > > - *Smart Sensors – *in 2.0 or 2.1
>>> > >> > > - AIP-17
>>> > >> > > <
>>> > >> > >
>>> > >> >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
>>> > >> > > >
>>> > >> > > |
>>> > >> > > PR: https://github.com/apache/airflow/pull/5499
>>> > >> > > - We have not come to a conclusion yet on whether this
>>> should
>>> > be
>>> > >> > > included in 2.0 or not. The majority is towards adding it
>>> in
>>> > 2.0
>>> > >> > (as
>>> > >> > > it
>>> > >> > > supports Airflow 2.0's Scalability story) and marking it
>>> as
>>> > >> > > *experimental*.
>>> > >> > > - There were some questions raised around supporting this
>>> new
>>> > >> > > feature. So we decided that *everyone would take a look at
>>> > the PR
>>> > >> > > itself and we will spend a few minutes in the next
>>> meeting to
>>> > >> > decide
>>> > >> > > whether it is 2.0 or not*.
>>> > >> > > - *Simplification of KubernetesExecutor /
>>> KubernetesPodOperator*
>>> > >> > > - PR: https://github.com/apache/airflow/pull/10393
>>> > >> > > - This will be part of *Airflow 2.0*
>>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
>>> > >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467
>>> |
>>> > Design
>>> > >> > > Doc:
>>> > >> > >
>>> > >> > >
>>> > >> >
>>> >
>>> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
>>> > >> > > - *Scope*:
>>> > >> > > - Users bash script won’t be included but anything in
>>> the
>>> > core
>>> > >> > > Airflow would be covered
>>> > >> > > -
>>> > >> > >
>>> > >> > > *DAG Definitions*:
>>> > >> > > - Changes in Path for contrib to Providers packages
>>> > >> > > - DAG Interfaces: changes in arguments of a DAG /
>>> > >> > BaseOperator
>>> > >> > > - *Configurations*:
>>> > >> > > - Option to auto-replace deprecated configs with new
>>> > options
>>> > >> > > - *Run-time Core items*:
>>> > >> > > - Changes like "Connection type can't be null". The
>>> > >> > > upgrade-check should at least shown warning if it
>>> can't
>>> > >> > > provide option to
>>> > >> > > detect the type.
>>> > >> > > - *CLI refactor is out-of-scope*
>>> > >> > > - Automatic refactor is *out-of-scope* as it is too
>>> > difficult
>>> > >> > > to cover all the cases in the Users bash scripts.
>>> > >> > > - This will be covered by docs or by showing
>>> warnings
>>> > via the
>>> > >> > > upgrade-check command
>>> > >> > > - *Experimental API to New API refactor is
>>> out-of-scope*
>>> > (will
>>> > >> > be
>>> > >> > > covered by Migration docs)
>>> > >> > > - We agreed that the airflow upgrade-check command *needs
>>> to
>>> > be
>>> > >> > > available in the last release before Airflow 2.0* (1.10.x
>>> or
>>> > >> > 1.11.x)
>>> > >> > > - Potential problems with time-consuming DB Migration were
>>> also
>>> > >> > > discussed. If we identify such a DB Migration (example the
>>> one
>>> > >> > involving
>>> > >> > > TaskInstance table) should be noted separately in
>>> Updating.md to
>>> > >> > > provide a
>>> > >> > > warning to the users.
>>> > >> > > - *DEV Calls Feedback*
>>> > >> > > - We agreed on having *Weekly calls from 7 September
>>> onwards*
>>> > >> > > - Calls will start with a 5-min reviewing the progress
>>> from
>>> > the
>>> > >> > last
>>> > >> > > call towards 2.0
>>> > >> > > - *Process*
>>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
>>> > >> > > - Changelog:
>>> > >> > > - The current way of Changelog is OK. We don't need
>>> further
>>> > >> > > categorization like Webserver, Scheduler etc.
>>> > >> > > - Separate Changelog would be created for Providers
>>> > Packages
>>> > >> > > - We need to figure a way to tag/label PRs & Issues
>>> with
>>> > correct
>>> > >> > > categories. Some options that were discussed were:
>>> > >> > > - Adding labels on the PRs & Issues via Bot
>>> > >> > > - A field in PR template for PR authors to add, the
>>> bot
>>> > would
>>> > >> > > then read the field which would be used to label
>>> the PR
>>> > >> > > - Add rules, for example Committers needs to add
>>> > appropriate
>>> > >> > > labels to the PR before merging it. We could have a
>>> > >> > > scheduled Github
>>> > >> > > Actions workflow that would fail if it finds PRs
>>> without
>>> > >> > > labels.
>>> > >> > >
>>> > >> > > *Things to Discuss Next*
>>> > >> > >
>>> > >> > > - *7 September*
>>> > >> > > - Progress, Current Work & Discussions
>>> > >> > > - API
>>> > >> > > - Providers Packages
>>> > >> > > - Discuss open questions
>>> > >> > > - Improvements to SubDags / Concept of TaskGroup
>>> > >> > > - AIP-34 <
>>> https://github.com/apache/airflow/pull/10153>
>>> > >> > > - *14 September*
>>> > >> > > - Process:
>>> > >> > > - When should we defer the in-scope items to post-2.0
>>> > >> > > - Completion by a date?
>>> > >> > > - Progress by a date?
>>> > >> > > - Progress, Current Work & Discussions
>>> > >> > > - Scheduler HA
>>> > >> > > - Docs Improvements
>>> > >> > > - Helm Chart
>>> > >> > > - Discuss the issue with sources
>>> > >> > >
>>> > >> > >
>>> > >> > > Regards,
>>> > >> > > Kaxil
>>> > >> > >
>>> > >> >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > >
>>> > > Jarek Potiuk
>>> > > Polidea | Principal Software Engineer
>>> > >
>>> > > M: +48 660 796 129
>>> > >
>>> > >
>>> >
>>> >
>>> > --
>>> >
>>> > Jarek Potiuk
>>> > Polidea | Principal Software Engineer
>>> >
>>> > M: +48 660 796 129
>>> >
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
Just a short reminder - for some more comments/review on the "PIP package
model of Airflow 2.0" doc
https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit#
I've added one small addition - in this model we want to make sure that
there are no dependencies of core packages on any of the providers - we do
not run such checks yet but it's easy to add.
J
On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Cool!
>
> If you have comments on particular sections/paragraphs - it's easier to
> keep track of it and respond in the doc. If you have some general
> statements, and some summary of your thinking after the review - it's best
> to respond to the email :)
>
> I am ok with both and will aggregate it eventually.
>
> J.
>
>
> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io> wrote:
>
>> Jarek,
>>
>> Thank you, this is very helpful.
>> I assume that you would like comments in the document itself?
>> Or, would you like them in email?
>>
>> Best regards,
>> Vikram
>>
>>
>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> > As promised during the last call I prepared the proposal on how we can
>> > approach the package model for Airflow 2.0 - including the "Provider
>> > Packages" approach.
>> >
>> > https://s.apache.org/airflow-2-0-package-model
>> >
>> > I would like to discuss it at our next meeting on Monday. I'd love to
>> > hear your comments.
>> >
>> > J.
>> >
>> >
>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <Jarek.Potiuk@polidea.com
>> >
>> > wrote:
>> > >
>> > > +1 Kevin on the call :).
>> > >
>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com>
>> wrote:
>> > >>
>> > >> Thanks Kevin, Looking forward to see you on the next call.
>> > >>
>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
>> > >>
>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to join at
>> > least the
>> > >> > next meeting trying to see if we can provide more perspectives on
>> > >> > SmartSensor and anything else we can help.
>> > >> >
>> > >> >
>> > >> > Cheers,
>> > >> > Kevin Y
>> > >> >
>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com>
>> > wrote:
>> > >> >
>> > >> > > Hi all,
>> > >> > >
>> > >> > > I have created a document to summarize the discussion from our
>> > second dev
>> > >> > > call for Airflow 2.0.
>> > >> > >
>> > >> > > Thank you all who joined the call.
>> > >> > >
>> > >> > > *Doc Link*:
>> > >> > >
>> > >> > >
>> > >> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
>> > <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>> >
>> > >> > <
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>> > >
>> > >> > > <
>> > >> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>> > >> > >
>> > >> > >
>> > >> > > To all those who attended, can you please double-check and add
>> if I
>> > have
>> > >> > > missed anything?
>> > >> > >
>> > >> > > To all those who didn't join, if you disagree to anything in the
>> > Summary
>> > >> > > please voice your opinion.
>> > >> > >
>> > >> > > Including the Summary here too (might potentially break
>> formatting):
>> > >> > >
>> > >> > > *Key Decisions*
>> > >> > >
>> > >> > > - *Smart Sensors – *in 2.0 or 2.1
>> > >> > > - AIP-17
>> > >> > > <
>> > >> > >
>> > >> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
>> > >> > > >
>> > >> > > |
>> > >> > > PR: https://github.com/apache/airflow/pull/5499
>> > >> > > - We have not come to a conclusion yet on whether this
>> should
>> > be
>> > >> > > included in 2.0 or not. The majority is towards adding it
>> in
>> > 2.0
>> > >> > (as
>> > >> > > it
>> > >> > > supports Airflow 2.0's Scalability story) and marking it as
>> > >> > > *experimental*.
>> > >> > > - There were some questions raised around supporting this
>> new
>> > >> > > feature. So we decided that *everyone would take a look at
>> > the PR
>> > >> > > itself and we will spend a few minutes in the next meeting
>> to
>> > >> > decide
>> > >> > > whether it is 2.0 or not*.
>> > >> > > - *Simplification of KubernetesExecutor /
>> KubernetesPodOperator*
>> > >> > > - PR: https://github.com/apache/airflow/pull/10393
>> > >> > > - This will be part of *Airflow 2.0*
>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
>> > >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467
>> |
>> > Design
>> > >> > > Doc:
>> > >> > >
>> > >> > >
>> > >> >
>> >
>> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
>> > >> > > - *Scope*:
>> > >> > > - Users bash script won’t be included but anything in
>> the
>> > core
>> > >> > > Airflow would be covered
>> > >> > > -
>> > >> > >
>> > >> > > *DAG Definitions*:
>> > >> > > - Changes in Path for contrib to Providers packages
>> > >> > > - DAG Interfaces: changes in arguments of a DAG /
>> > >> > BaseOperator
>> > >> > > - *Configurations*:
>> > >> > > - Option to auto-replace deprecated configs with new
>> > options
>> > >> > > - *Run-time Core items*:
>> > >> > > - Changes like "Connection type can't be null". The
>> > >> > > upgrade-check should at least shown warning if it
>> can't
>> > >> > > provide option to
>> > >> > > detect the type.
>> > >> > > - *CLI refactor is out-of-scope*
>> > >> > > - Automatic refactor is *out-of-scope* as it is too
>> > difficult
>> > >> > > to cover all the cases in the Users bash scripts.
>> > >> > > - This will be covered by docs or by showing warnings
>> > via the
>> > >> > > upgrade-check command
>> > >> > > - *Experimental API to New API refactor is out-of-scope*
>> > (will
>> > >> > be
>> > >> > > covered by Migration docs)
>> > >> > > - We agreed that the airflow upgrade-check command *needs
>> to
>> > be
>> > >> > > available in the last release before Airflow 2.0* (1.10.x
>> or
>> > >> > 1.11.x)
>> > >> > > - Potential problems with time-consuming DB Migration were
>> also
>> > >> > > discussed. If we identify such a DB Migration (example the one
>> > >> > involving
>> > >> > > TaskInstance table) should be noted separately in Updating.md
>> to
>> > >> > > provide a
>> > >> > > warning to the users.
>> > >> > > - *DEV Calls Feedback*
>> > >> > > - We agreed on having *Weekly calls from 7 September
>> onwards*
>> > >> > > - Calls will start with a 5-min reviewing the progress from
>> > the
>> > >> > last
>> > >> > > call towards 2.0
>> > >> > > - *Process*
>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
>> > >> > > - Changelog:
>> > >> > > - The current way of Changelog is OK. We don't need
>> further
>> > >> > > categorization like Webserver, Scheduler etc.
>> > >> > > - Separate Changelog would be created for Providers
>> > Packages
>> > >> > > - We need to figure a way to tag/label PRs & Issues with
>> > correct
>> > >> > > categories. Some options that were discussed were:
>> > >> > > - Adding labels on the PRs & Issues via Bot
>> > >> > > - A field in PR template for PR authors to add, the
>> bot
>> > would
>> > >> > > then read the field which would be used to label the
>> PR
>> > >> > > - Add rules, for example Committers needs to add
>> > appropriate
>> > >> > > labels to the PR before merging it. We could have a
>> > >> > > scheduled Github
>> > >> > > Actions workflow that would fail if it finds PRs
>> without
>> > >> > > labels.
>> > >> > >
>> > >> > > *Things to Discuss Next*
>> > >> > >
>> > >> > > - *7 September*
>> > >> > > - Progress, Current Work & Discussions
>> > >> > > - API
>> > >> > > - Providers Packages
>> > >> > > - Discuss open questions
>> > >> > > - Improvements to SubDags / Concept of TaskGroup
>> > >> > > - AIP-34 <
>> https://github.com/apache/airflow/pull/10153>
>> > >> > > - *14 September*
>> > >> > > - Process:
>> > >> > > - When should we defer the in-scope items to post-2.0
>> > >> > > - Completion by a date?
>> > >> > > - Progress by a date?
>> > >> > > - Progress, Current Work & Discussions
>> > >> > > - Scheduler HA
>> > >> > > - Docs Improvements
>> > >> > > - Helm Chart
>> > >> > > - Discuss the issue with sources
>> > >> > >
>> > >> > >
>> > >> > > Regards,
>> > >> > > Kaxil
>> > >> > >
>> > >> >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Jarek Potiuk
>> > > Polidea | Principal Software Engineer
>> > >
>> > > M: +48 660 796 129
>> > >
>> > >
>> >
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea | Principal Software Engineer
>> >
>> > M: +48 660 796 129
>> >
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
Cool!
If you have comments on particular sections/paragraphs - it's easier to
keep track of it and respond in the doc. If you have some general
statements, and some summary of your thinking after the review - it's best
to respond to the email :)
I am ok with both and will aggregate it eventually.
J.
On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io> wrote:
> Jarek,
>
> Thank you, this is very helpful.
> I assume that you would like comments in the document itself?
> Or, would you like them in email?
>
> Best regards,
> Vikram
>
>
> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > As promised during the last call I prepared the proposal on how we can
> > approach the package model for Airflow 2.0 - including the "Provider
> > Packages" approach.
> >
> > https://s.apache.org/airflow-2-0-package-model
> >
> > I would like to discuss it at our next meeting on Monday. I'd love to
> > hear your comments.
> >
> > J.
> >
> >
> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > >
> > > +1 Kevin on the call :).
> > >
> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >>
> > >> Thanks Kevin, Looking forward to see you on the next call.
> > >>
> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
> > >>
> > >> > Thank you Kaxil, this is extremely helpful. We'll try to join at
> > least the
> > >> > next meeting trying to see if we can provide more perspectives on
> > >> > SmartSensor and anything else we can help.
> > >> >
> > >> >
> > >> > Cheers,
> > >> > Kevin Y
> > >> >
> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > >> >
> > >> > > Hi all,
> > >> > >
> > >> > > I have created a document to summarize the discussion from our
> > second dev
> > >> > > call for Airflow 2.0.
> > >> > >
> > >> > > Thank you all who joined the call.
> > >> > >
> > >> > > *Doc Link*:
> > >> > >
> > >> > >
> > >> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >
> > >> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> > >
> > >> > > <
> > >> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> > >> > >
> > >> > >
> > >> > > To all those who attended, can you please double-check and add if
> I
> > have
> > >> > > missed anything?
> > >> > >
> > >> > > To all those who didn't join, if you disagree to anything in the
> > Summary
> > >> > > please voice your opinion.
> > >> > >
> > >> > > Including the Summary here too (might potentially break
> formatting):
> > >> > >
> > >> > > *Key Decisions*
> > >> > >
> > >> > > - *Smart Sensors – *in 2.0 or 2.1
> > >> > > - AIP-17
> > >> > > <
> > >> > >
> > >> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
> > >> > > >
> > >> > > |
> > >> > > PR: https://github.com/apache/airflow/pull/5499
> > >> > > - We have not come to a conclusion yet on whether this
> should
> > be
> > >> > > included in 2.0 or not. The majority is towards adding it in
> > 2.0
> > >> > (as
> > >> > > it
> > >> > > supports Airflow 2.0's Scalability story) and marking it as
> > >> > > *experimental*.
> > >> > > - There were some questions raised around supporting this
> new
> > >> > > feature. So we decided that *everyone would take a look at
> > the PR
> > >> > > itself and we will spend a few minutes in the next meeting
> to
> > >> > decide
> > >> > > whether it is 2.0 or not*.
> > >> > > - *Simplification of KubernetesExecutor /
> KubernetesPodOperator*
> > >> > > - PR: https://github.com/apache/airflow/pull/10393
> > >> > > - This will be part of *Airflow 2.0*
> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
> > >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 |
> > Design
> > >> > > Doc:
> > >> > >
> > >> > >
> > >> >
> >
> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
> > >> > > - *Scope*:
> > >> > > - Users bash script won’t be included but anything in the
> > core
> > >> > > Airflow would be covered
> > >> > > -
> > >> > >
> > >> > > *DAG Definitions*:
> > >> > > - Changes in Path for contrib to Providers packages
> > >> > > - DAG Interfaces: changes in arguments of a DAG /
> > >> > BaseOperator
> > >> > > - *Configurations*:
> > >> > > - Option to auto-replace deprecated configs with new
> > options
> > >> > > - *Run-time Core items*:
> > >> > > - Changes like "Connection type can't be null". The
> > >> > > upgrade-check should at least shown warning if it
> can't
> > >> > > provide option to
> > >> > > detect the type.
> > >> > > - *CLI refactor is out-of-scope*
> > >> > > - Automatic refactor is *out-of-scope* as it is too
> > difficult
> > >> > > to cover all the cases in the Users bash scripts.
> > >> > > - This will be covered by docs or by showing warnings
> > via the
> > >> > > upgrade-check command
> > >> > > - *Experimental API to New API refactor is out-of-scope*
> > (will
> > >> > be
> > >> > > covered by Migration docs)
> > >> > > - We agreed that the airflow upgrade-check command *needs to
> > be
> > >> > > available in the last release before Airflow 2.0* (1.10.x or
> > >> > 1.11.x)
> > >> > > - Potential problems with time-consuming DB Migration were also
> > >> > > discussed. If we identify such a DB Migration (example the one
> > >> > involving
> > >> > > TaskInstance table) should be noted separately in Updating.md
> to
> > >> > > provide a
> > >> > > warning to the users.
> > >> > > - *DEV Calls Feedback*
> > >> > > - We agreed on having *Weekly calls from 7 September
> onwards*
> > >> > > - Calls will start with a 5-min reviewing the progress from
> > the
> > >> > last
> > >> > > call towards 2.0
> > >> > > - *Process*
> > >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
> > >> > > - Changelog:
> > >> > > - The current way of Changelog is OK. We don't need
> further
> > >> > > categorization like Webserver, Scheduler etc.
> > >> > > - Separate Changelog would be created for Providers
> > Packages
> > >> > > - We need to figure a way to tag/label PRs & Issues with
> > correct
> > >> > > categories. Some options that were discussed were:
> > >> > > - Adding labels on the PRs & Issues via Bot
> > >> > > - A field in PR template for PR authors to add, the
> bot
> > would
> > >> > > then read the field which would be used to label the
> PR
> > >> > > - Add rules, for example Committers needs to add
> > appropriate
> > >> > > labels to the PR before merging it. We could have a
> > >> > > scheduled Github
> > >> > > Actions workflow that would fail if it finds PRs
> without
> > >> > > labels.
> > >> > >
> > >> > > *Things to Discuss Next*
> > >> > >
> > >> > > - *7 September*
> > >> > > - Progress, Current Work & Discussions
> > >> > > - API
> > >> > > - Providers Packages
> > >> > > - Discuss open questions
> > >> > > - Improvements to SubDags / Concept of TaskGroup
> > >> > > - AIP-34 <
> https://github.com/apache/airflow/pull/10153>
> > >> > > - *14 September*
> > >> > > - Process:
> > >> > > - When should we defer the in-scope items to post-2.0
> > >> > > - Completion by a date?
> > >> > > - Progress by a date?
> > >> > > - Progress, Current Work & Discussions
> > >> > > - Scheduler HA
> > >> > > - Docs Improvements
> > >> > > - Helm Chart
> > >> > > - Discuss the issue with sources
> > >> > >
> > >> > >
> > >> > > Regards,
> > >> > > Kaxil
> > >> > >
> > >> >
> > >
> > >
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea | Principal Software Engineer
> > >
> > > M: +48 660 796 129
> > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Vikram Koka <vi...@astronomer.io>.
Jarek,
Thank you, this is very helpful.
I assume that you would like comments in the document itself?
Or, would you like them in email?
Best regards,
Vikram
On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> As promised during the last call I prepared the proposal on how we can
> approach the package model for Airflow 2.0 - including the "Provider
> Packages" approach.
>
> https://s.apache.org/airflow-2-0-package-model
>
> I would like to discuss it at our next meeting on Monday. I'd love to
> hear your comments.
>
> J.
>
>
> On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >
> > +1 Kevin on the call :).
> >
> > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com> wrote:
> >>
> >> Thanks Kevin, Looking forward to see you on the next call.
> >>
> >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
> >>
> >> > Thank you Kaxil, this is extremely helpful. We'll try to join at
> least the
> >> > next meeting trying to see if we can provide more perspectives on
> >> > SmartSensor and anything else we can help.
> >> >
> >> >
> >> > Cheers,
> >> > Kevin Y
> >> >
> >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com>
> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I have created a document to summarize the discussion from our
> second dev
> >> > > call for Airflow 2.0.
> >> > >
> >> > > Thank you all who joined the call.
> >> > >
> >> > > *Doc Link*:
> >> > >
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
> >> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >
> >> > > <
> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >> > >
> >> > >
> >> > > To all those who attended, can you please double-check and add if I
> have
> >> > > missed anything?
> >> > >
> >> > > To all those who didn't join, if you disagree to anything in the
> Summary
> >> > > please voice your opinion.
> >> > >
> >> > > Including the Summary here too (might potentially break formatting):
> >> > >
> >> > > *Key Decisions*
> >> > >
> >> > > - *Smart Sensors – *in 2.0 or 2.1
> >> > > - AIP-17
> >> > > <
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
> >> > > >
> >> > > |
> >> > > PR: https://github.com/apache/airflow/pull/5499
> >> > > - We have not come to a conclusion yet on whether this should
> be
> >> > > included in 2.0 or not. The majority is towards adding it in
> 2.0
> >> > (as
> >> > > it
> >> > > supports Airflow 2.0's Scalability story) and marking it as
> >> > > *experimental*.
> >> > > - There were some questions raised around supporting this new
> >> > > feature. So we decided that *everyone would take a look at
> the PR
> >> > > itself and we will spend a few minutes in the next meeting to
> >> > decide
> >> > > whether it is 2.0 or not*.
> >> > > - *Simplification of KubernetesExecutor / KubernetesPodOperator*
> >> > > - PR: https://github.com/apache/airflow/pull/10393
> >> > > - This will be part of *Airflow 2.0*
> >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
> >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 |
> Design
> >> > > Doc:
> >> > >
> >> > >
> >> >
> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
> >> > > - *Scope*:
> >> > > - Users bash script won’t be included but anything in the
> core
> >> > > Airflow would be covered
> >> > > -
> >> > >
> >> > > *DAG Definitions*:
> >> > > - Changes in Path for contrib to Providers packages
> >> > > - DAG Interfaces: changes in arguments of a DAG /
> >> > BaseOperator
> >> > > - *Configurations*:
> >> > > - Option to auto-replace deprecated configs with new
> options
> >> > > - *Run-time Core items*:
> >> > > - Changes like "Connection type can't be null". The
> >> > > upgrade-check should at least shown warning if it can't
> >> > > provide option to
> >> > > detect the type.
> >> > > - *CLI refactor is out-of-scope*
> >> > > - Automatic refactor is *out-of-scope* as it is too
> difficult
> >> > > to cover all the cases in the Users bash scripts.
> >> > > - This will be covered by docs or by showing warnings
> via the
> >> > > upgrade-check command
> >> > > - *Experimental API to New API refactor is out-of-scope*
> (will
> >> > be
> >> > > covered by Migration docs)
> >> > > - We agreed that the airflow upgrade-check command *needs to
> be
> >> > > available in the last release before Airflow 2.0* (1.10.x or
> >> > 1.11.x)
> >> > > - Potential problems with time-consuming DB Migration were also
> >> > > discussed. If we identify such a DB Migration (example the one
> >> > involving
> >> > > TaskInstance table) should be noted separately in Updating.md to
> >> > > provide a
> >> > > warning to the users.
> >> > > - *DEV Calls Feedback*
> >> > > - We agreed on having *Weekly calls from 7 September onwards*
> >> > > - Calls will start with a 5-min reviewing the progress from
> the
> >> > last
> >> > > call towards 2.0
> >> > > - *Process*
> >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
> >> > > - Changelog:
> >> > > - The current way of Changelog is OK. We don't need further
> >> > > categorization like Webserver, Scheduler etc.
> >> > > - Separate Changelog would be created for Providers
> Packages
> >> > > - We need to figure a way to tag/label PRs & Issues with
> correct
> >> > > categories. Some options that were discussed were:
> >> > > - Adding labels on the PRs & Issues via Bot
> >> > > - A field in PR template for PR authors to add, the bot
> would
> >> > > then read the field which would be used to label the PR
> >> > > - Add rules, for example Committers needs to add
> appropriate
> >> > > labels to the PR before merging it. We could have a
> >> > > scheduled Github
> >> > > Actions workflow that would fail if it finds PRs without
> >> > > labels.
> >> > >
> >> > > *Things to Discuss Next*
> >> > >
> >> > > - *7 September*
> >> > > - Progress, Current Work & Discussions
> >> > > - API
> >> > > - Providers Packages
> >> > > - Discuss open questions
> >> > > - Improvements to SubDags / Concept of TaskGroup
> >> > > - AIP-34 <https://github.com/apache/airflow/pull/10153>
> >> > > - *14 September*
> >> > > - Process:
> >> > > - When should we defer the in-scope items to post-2.0
> >> > > - Completion by a date?
> >> > > - Progress by a date?
> >> > > - Progress, Current Work & Discussions
> >> > > - Scheduler HA
> >> > > - Docs Improvements
> >> > > - Helm Chart
> >> > > - Discuss the issue with sources
> >> > >
> >> > >
> >> > > Regards,
> >> > > Kaxil
> >> > >
> >> >
> >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
> >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>