You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@gmail.com> on 2020/08/25 23:27:49 UTC
[Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Hi all,
I have created a document to summarize the discussion from our second dev
call for Airflow 2.0.
Thank you all who joined the call.
*Doc Link*:
https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
To all those who attended, can you please double-check and add if I have
missed anything?
To all those who didn't join, if you disagree to anything in the Summary
please voice your opinion.
Including the Summary here too (might potentially break formatting):
*Key Decisions*
- *Smart Sensors – *in 2.0 or 2.1
- AIP-17
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor>
|
PR: https://github.com/apache/airflow/pull/5499
- We have not come to a conclusion yet on whether this should be
included in 2.0 or not. The majority is towards adding it in 2.0 (as it
supports Airflow 2.0's Scalability story) and marking it as
*experimental*.
- There were some questions raised around supporting this new
feature. So we decided that *everyone would take a look at the PR
itself and we will spend a few minutes in the next meeting to decide
whether it is 2.0 or not*.
- *Simplification of KubernetesExecutor / KubernetesPodOperator*
- PR: https://github.com/apache/airflow/pull/10393
- This will be part of *Airflow 2.0*
- *Airflow Upgrade Check* (airflow upgrade-check)* command *
- WIP PR: PR: https://github.com/apache/airflow/pull/9467 | Design
Doc:
https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
- *Scope*:
- Users bash script won’t be included but anything in the core
Airflow would be covered
-
*DAG Definitions*:
- Changes in Path for contrib to Providers packages
- DAG Interfaces: changes in arguments of a DAG / BaseOperator
- *Configurations*:
- Option to auto-replace deprecated configs with new options
- *Run-time Core items*:
- Changes like "Connection type can't be null". The
upgrade-check should at least shown warning if it can't
provide option to
detect the type.
- *CLI refactor is out-of-scope*
- Automatic refactor is *out-of-scope* as it is too difficult
to cover all the cases in the Users bash scripts.
- This will be covered by docs or by showing warnings via the
upgrade-check command
- *Experimental API to New API refactor is out-of-scope* (will be
covered by Migration docs)
- We agreed that the airflow upgrade-check command *needs to be
available in the last release before Airflow 2.0* (1.10.x or 1.11.x)
- Potential problems with time-consuming DB Migration were also
discussed. If we identify such a DB Migration (example the one involving
TaskInstance table) should be noted separately in Updating.md to provide a
warning to the users.
- *DEV Calls Feedback*
- We agreed on having *Weekly calls from 7 September onwards*
- Calls will start with a 5-min reviewing the progress from the last
call towards 2.0
- *Process*
- A *2.0.0-test* branch will be created on 10 Sep 2020
- Changelog:
- The current way of Changelog is OK. We don't need further
categorization like Webserver, Scheduler etc.
- Separate Changelog would be created for Providers Packages
- We need to figure a way to tag/label PRs & Issues with correct
categories. Some options that were discussed were:
- Adding labels on the PRs & Issues via Bot
- A field in PR template for PR authors to add, the bot would
then read the field which would be used to label the PR
- Add rules, for example Committers needs to add appropriate
labels to the PR before merging it. We could have a
scheduled Github
Actions workflow that would fail if it finds PRs without labels.
*Things to Discuss Next*
- *7 September*
- Progress, Current Work & Discussions
- API
- Providers Packages
- Discuss open questions
- Improvements to SubDags / Concept of TaskGroup
- AIP-34 <https://github.com/apache/airflow/pull/10153>
- *14 September*
- Process:
- When should we defer the in-scope items to post-2.0
- Completion by a date?
- Progress by a date?
- Progress, Current Work & Discussions
- Scheduler HA
- Docs Improvements
- Helm Chart
- Discuss the issue with sources
Regards,
Kaxil
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
Yep. Thanks, Ash - indeed that is closely related and if do this, I think
9506 should be fixed together. I added it to the doc.
I also resolved some of the questions and comments in the doc and added a
short "Alternative" - i.e. what is the alternative - monolith Airflow
release as we did so far.
I think at the call today we should start from this "alternative" - because
maybe the gains that we have by releasing separate packages are not as big
as the complexity we are adding.
I am a bit on the fence on that one myself - I am not 100 % sure if the
gains are really as big, I would love to hear what others think about it :)
J.
On Sat, Sep 5, 2020 at 11:28 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> Very closely related issue https://github.com/apache/airflow/issues/9506
>
> On 5 September 2020 08:01:11 BST, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >And we have a new addition from Kamil about the need to extend slightly
> >plugin mechanism to be able to cover dynamically "Connections",
> >"Connection
> >Form" and "Extra Links" - those are indeed the "core -> Providers"
> >dependencies that we still have.
> >
> >They seem to be easy to handle by making providers "plugins" and
> >extending
> >the plugin mechanism a bit. Thanks Kamil for the thoughtful input!
> >
> >On Fri, Sep 4, 2020 at 8:08 PM Jarek Potiuk <Ja...@polidea.com>
> >wrote:
> >
> >> Just a short reminder - for some more comments/review on the "PIP
> >package
> >> model of Airflow 2.0" doc
> >>
> >>
> >
> https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit#
> >>
> >> I've added one small addition - in this model we want to make sure
> >that
> >> there are no dependencies of core packages on any of the providers -
> >we do
> >> not run such checks yet but it's easy to add.
> >>
> >> J
> >>
> >>
> >> On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk
> ><Ja...@polidea.com>
> >> wrote:
> >>
> >>> Cool!
> >>>
> >>> If you have comments on particular sections/paragraphs - it's easier
> >to
> >>> keep track of it and respond in the doc. If you have some general
> >>> statements, and some summary of your thinking after the review -
> >it's best
> >>> to respond to the email :)
> >>>
> >>> I am ok with both and will aggregate it eventually.
> >>>
> >>> J.
> >>>
> >>>
> >>> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io>
> >wrote:
> >>>
> >>>> Jarek,
> >>>>
> >>>> Thank you, this is very helpful.
> >>>> I assume that you would like comments in the document itself?
> >>>> Or, would you like them in email?
> >>>>
> >>>> Best regards,
> >>>> Vikram
> >>>>
> >>>>
> >>>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk
> ><Ja...@polidea.com>
> >>>> wrote:
> >>>>
> >>>> > As promised during the last call I prepared the proposal on how
> >we can
> >>>> > approach the package model for Airflow 2.0 - including the
> >"Provider
> >>>> > Packages" approach.
> >>>> >
> >>>> > https://s.apache.org/airflow-2-0-package-model
> >>>> >
> >>>> > I would like to discuss it at our next meeting on Monday. I'd
> >love to
> >>>> > hear your comments.
> >>>> >
> >>>> > J.
> >>>> >
> >>>> >
> >>>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <
> >>>> Jarek.Potiuk@polidea.com>
> >>>> > wrote:
> >>>> > >
> >>>> > > +1 Kevin on the call :).
> >>>> > >
> >>>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik
> ><ka...@gmail.com>
> >>>> wrote:
> >>>> > >>
> >>>> > >> Thanks Kevin, Looking forward to see you on the next call.
> >>>> > >>
> >>>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com>
> >wrote:
> >>>> > >>
> >>>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to
> >join at
> >>>> > least the
> >>>> > >> > next meeting trying to see if we can provide more
> >perspectives on
> >>>> > >> > SmartSensor and anything else we can help.
> >>>> > >> >
> >>>> > >> >
> >>>> > >> > Cheers,
> >>>> > >> > Kevin Y
> >>>> > >> >
> >>>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik
> ><ka...@gmail.com>
> >>>> > wrote:
> >>>> > >> >
> >>>> > >> > > Hi all,
> >>>> > >> > >
> >>>> > >> > > I have created a document to summarize the discussion from
> >our
> >>>> > second dev
> >>>> > >> > > call for Airflow 2.0.
> >>>> > >> > >
> >>>> > >> > > Thank you all who joined the call.
> >>>> > >> > >
> >>>> > >> > > *Doc Link*:
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> >
> >>>> >
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
> >>>>
> ><
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >
> >>>> > <
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >>>> >
> >>>> > >> > <
> >>>> >
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >>>> > >
> >>>> > >> > > <
> >>>> > >> >
> >>>> >
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > > To all those who attended, can you please double-check and
> >add
> >>>> if I
> >>>> > have
> >>>> > >> > > missed anything?
> >>>> > >> > >
> >>>> > >> > > To all those who didn't join, if you disagree to anything
> >in the
> >>>> > Summary
> >>>> > >> > > please voice your opinion.
> >>>> > >> > >
> >>>> > >> > > Including the Summary here too (might potentially break
> >>>> formatting):
> >>>> > >> > >
> >>>> > >> > > *Key Decisions*
> >>>> > >> > >
> >>>> > >> > > - *Smart Sensors – *in 2.0 or 2.1
> >>>> > >> > > - AIP-17
> >>>> > >> > > <
> >>>> > >> > >
> >>>> > >> >
> >>>> >
> >>>>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
> >>>> > >> > > >
> >>>> > >> > > |
> >>>> > >> > > PR: https://github.com/apache/airflow/pull/5499
> >>>> > >> > > - We have not come to a conclusion yet on whether
> >this
> >>>> should
> >>>> > be
> >>>> > >> > > included in 2.0 or not. The majority is towards
> >adding it
> >>>> in
> >>>> > 2.0
> >>>> > >> > (as
> >>>> > >> > > it
> >>>> > >> > > supports Airflow 2.0's Scalability story) and
> >marking it
> >>>> as
> >>>> > >> > > *experimental*.
> >>>> > >> > > - There were some questions raised around supporting
> >this
> >>>> new
> >>>> > >> > > feature. So we decided that *everyone would take a
> >look at
> >>>> > the PR
> >>>> > >> > > itself and we will spend a few minutes in the next
> >>>> meeting to
> >>>> > >> > decide
> >>>> > >> > > whether it is 2.0 or not*.
> >>>> > >> > > - *Simplification of KubernetesExecutor /
> >>>> KubernetesPodOperator*
> >>>> > >> > > - PR: https://github.com/apache/airflow/pull/10393
> >>>> > >> > > - This will be part of *Airflow 2.0*
> >>>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)*
> >command *
> >>>> > >> > > - WIP PR: PR:
> >https://github.com/apache/airflow/pull/9467
> >>>> |
> >>>> > Design
> >>>> > >> > > Doc:
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> >
> >>>> >
> >>>>
> >
> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
> >>>> > >> > > - *Scope*:
> >>>> > >> > > - Users bash script won’t be included but
> >anything in
> >>>> the
> >>>> > core
> >>>> > >> > > Airflow would be covered
> >>>> > >> > > -
> >>>> > >> > >
> >>>> > >> > > *DAG Definitions*:
> >>>> > >> > > - Changes in Path for contrib to Providers
> >packages
> >>>> > >> > > - DAG Interfaces: changes in arguments of a
> >DAG /
> >>>> > >> > BaseOperator
> >>>> > >> > > - *Configurations*:
> >>>> > >> > > - Option to auto-replace deprecated configs
> >with new
> >>>> > options
> >>>> > >> > > - *Run-time Core items*:
> >>>> > >> > > - Changes like "Connection type can't be
> >null". The
> >>>> > >> > > upgrade-check should at least shown warning if
> >it
> >>>> can't
> >>>> > >> > > provide option to
> >>>> > >> > > detect the type.
> >>>> > >> > > - *CLI refactor is out-of-scope*
> >>>> > >> > > - Automatic refactor is *out-of-scope* as it
> >is too
> >>>> > difficult
> >>>> > >> > > to cover all the cases in the Users bash
> >scripts.
> >>>> > >> > > - This will be covered by docs or by showing
> >>>> warnings
> >>>> > via the
> >>>> > >> > > upgrade-check command
> >>>> > >> > > - *Experimental API to New API refactor is
> >>>> out-of-scope*
> >>>> > (will
> >>>> > >> > be
> >>>> > >> > > covered by Migration docs)
> >>>> > >> > > - We agreed that the airflow upgrade-check command
> >*needs
> >>>> to
> >>>> > be
> >>>> > >> > > available in the last release before Airflow 2.0*
> >(1.10.x
> >>>> or
> >>>> > >> > 1.11.x)
> >>>> > >> > > - Potential problems with time-consuming DB Migration
> >were
> >>>> also
> >>>> > >> > > discussed. If we identify such a DB Migration (example
> >the
> >>>> one
> >>>> > >> > involving
> >>>> > >> > > TaskInstance table) should be noted separately in
> >>>> Updating.md to
> >>>> > >> > > provide a
> >>>> > >> > > warning to the users.
> >>>> > >> > > - *DEV Calls Feedback*
> >>>> > >> > > - We agreed on having *Weekly calls from 7 September
> >>>> onwards*
> >>>> > >> > > - Calls will start with a 5-min reviewing the
> >progress
> >>>> from
> >>>> > the
> >>>> > >> > last
> >>>> > >> > > call towards 2.0
> >>>> > >> > > - *Process*
> >>>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep
> >2020
> >>>> > >> > > - Changelog:
> >>>> > >> > > - The current way of Changelog is OK. We don't
> >need
> >>>> further
> >>>> > >> > > categorization like Webserver, Scheduler etc.
> >>>> > >> > > - Separate Changelog would be created for
> >Providers
> >>>> > Packages
> >>>> > >> > > - We need to figure a way to tag/label PRs &
> >Issues
> >>>> with
> >>>> > correct
> >>>> > >> > > categories. Some options that were discussed
> >were:
> >>>> > >> > > - Adding labels on the PRs & Issues via Bot
> >>>> > >> > > - A field in PR template for PR authors to
> >add, the
> >>>> bot
> >>>> > would
> >>>> > >> > > then read the field which would be used to
> >label
> >>>> the PR
> >>>> > >> > > - Add rules, for example Committers needs to
> >add
> >>>> > appropriate
> >>>> > >> > > labels to the PR before merging it. We could
> >have a
> >>>> > >> > > scheduled Github
> >>>> > >> > > Actions workflow that would fail if it finds
> >PRs
> >>>> without
> >>>> > >> > > labels.
> >>>> > >> > >
> >>>> > >> > > *Things to Discuss Next*
> >>>> > >> > >
> >>>> > >> > > - *7 September*
> >>>> > >> > > - Progress, Current Work & Discussions
> >>>> > >> > > - API
> >>>> > >> > > - Providers Packages
> >>>> > >> > > - Discuss open questions
> >>>> > >> > > - Improvements to SubDags / Concept of TaskGroup
> >>>> > >> > > - AIP-34 <
> >>>> https://github.com/apache/airflow/pull/10153>
> >>>> > >> > > - *14 September*
> >>>> > >> > > - Process:
> >>>> > >> > > - When should we defer the in-scope items to
> >post-2.0
> >>>> > >> > > - Completion by a date?
> >>>> > >> > > - Progress by a date?
> >>>> > >> > > - Progress, Current Work & Discussions
> >>>> > >> > > - Scheduler HA
> >>>> > >> > > - Docs Improvements
> >>>> > >> > > - Helm Chart
> >>>> > >> > > - Discuss the issue with sources
> >>>> > >> > >
> >>>> > >> > >
> >>>> > >> > > Regards,
> >>>> > >> > > Kaxil
> >>>> > >> > >
> >>>> > >> >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > --
> >>>> > >
> >>>> > > Jarek Potiuk
> >>>> > > Polidea | Principal Software Engineer
> >>>> > >
> >>>> > > M: +48 660 796 129
> >>>> > >
> >>>> > >
> >>>> >
> >>>> >
> >>>> > --
> >>>> >
> >>>> > Jarek Potiuk
> >>>> > Polidea | Principal Software Engineer
> >>>> >
> >>>> > M: +48 660 796 129
> >>>> >
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>> Jarek Potiuk
> >>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>
> >>> M: +48 660 796 129 <+48660796129>
> >>> [image: Polidea] <https://www.polidea.com/>
> >>>
> >>>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >>
> >
> >--
> >
> >Jarek Potiuk
> >Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> >M: +48 660 796 129 <+48660796129>
> >[image: Polidea] <https://www.polidea.com/>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Ash Berlin-Taylor <as...@apache.org>.
Very closely related issue https://github.com/apache/airflow/issues/9506
On 5 September 2020 08:01:11 BST, Jarek Potiuk <Ja...@polidea.com> wrote:
>And we have a new addition from Kamil about the need to extend slightly
>plugin mechanism to be able to cover dynamically "Connections",
>"Connection
>Form" and "Extra Links" - those are indeed the "core -> Providers"
>dependencies that we still have.
>
>They seem to be easy to handle by making providers "plugins" and
>extending
>the plugin mechanism a bit. Thanks Kamil for the thoughtful input!
>
>On Fri, Sep 4, 2020 at 8:08 PM Jarek Potiuk <Ja...@polidea.com>
>wrote:
>
>> Just a short reminder - for some more comments/review on the "PIP
>package
>> model of Airflow 2.0" doc
>>
>>
>https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit#
>>
>> I've added one small addition - in this model we want to make sure
>that
>> there are no dependencies of core packages on any of the providers -
>we do
>> not run such checks yet but it's easy to add.
>>
>> J
>>
>>
>> On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk
><Ja...@polidea.com>
>> wrote:
>>
>>> Cool!
>>>
>>> If you have comments on particular sections/paragraphs - it's easier
>to
>>> keep track of it and respond in the doc. If you have some general
>>> statements, and some summary of your thinking after the review -
>it's best
>>> to respond to the email :)
>>>
>>> I am ok with both and will aggregate it eventually.
>>>
>>> J.
>>>
>>>
>>> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io>
>wrote:
>>>
>>>> Jarek,
>>>>
>>>> Thank you, this is very helpful.
>>>> I assume that you would like comments in the document itself?
>>>> Or, would you like them in email?
>>>>
>>>> Best regards,
>>>> Vikram
>>>>
>>>>
>>>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk
><Ja...@polidea.com>
>>>> wrote:
>>>>
>>>> > As promised during the last call I prepared the proposal on how
>we can
>>>> > approach the package model for Airflow 2.0 - including the
>"Provider
>>>> > Packages" approach.
>>>> >
>>>> > https://s.apache.org/airflow-2-0-package-model
>>>> >
>>>> > I would like to discuss it at our next meeting on Monday. I'd
>love to
>>>> > hear your comments.
>>>> >
>>>> > J.
>>>> >
>>>> >
>>>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <
>>>> Jarek.Potiuk@polidea.com>
>>>> > wrote:
>>>> > >
>>>> > > +1 Kevin on the call :).
>>>> > >
>>>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik
><ka...@gmail.com>
>>>> wrote:
>>>> > >>
>>>> > >> Thanks Kevin, Looking forward to see you on the next call.
>>>> > >>
>>>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com>
>wrote:
>>>> > >>
>>>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to
>join at
>>>> > least the
>>>> > >> > next meeting trying to see if we can provide more
>perspectives on
>>>> > >> > SmartSensor and anything else we can help.
>>>> > >> >
>>>> > >> >
>>>> > >> > Cheers,
>>>> > >> > Kevin Y
>>>> > >> >
>>>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik
><ka...@gmail.com>
>>>> > wrote:
>>>> > >> >
>>>> > >> > > Hi all,
>>>> > >> > >
>>>> > >> > > I have created a document to summarize the discussion from
>our
>>>> > second dev
>>>> > >> > > call for Airflow 2.0.
>>>> > >> > >
>>>> > >> > > Thank you all who joined the call.
>>>> > >> > >
>>>> > >> > > *Doc Link*:
>>>> > >> > >
>>>> > >> > >
>>>> > >> >
>>>> >
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
>>>>
><https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
>>>> > <
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>>> >
>>>> > >> > <
>>>> >
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>>> > >
>>>> > >> > > <
>>>> > >> >
>>>> >
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>>> > >> > >
>>>> > >> > >
>>>> > >> > > To all those who attended, can you please double-check and
>add
>>>> if I
>>>> > have
>>>> > >> > > missed anything?
>>>> > >> > >
>>>> > >> > > To all those who didn't join, if you disagree to anything
>in the
>>>> > Summary
>>>> > >> > > please voice your opinion.
>>>> > >> > >
>>>> > >> > > Including the Summary here too (might potentially break
>>>> formatting):
>>>> > >> > >
>>>> > >> > > *Key Decisions*
>>>> > >> > >
>>>> > >> > > - *Smart Sensors – *in 2.0 or 2.1
>>>> > >> > > - AIP-17
>>>> > >> > > <
>>>> > >> > >
>>>> > >> >
>>>> >
>>>>
>https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
>>>> > >> > > >
>>>> > >> > > |
>>>> > >> > > PR: https://github.com/apache/airflow/pull/5499
>>>> > >> > > - We have not come to a conclusion yet on whether
>this
>>>> should
>>>> > be
>>>> > >> > > included in 2.0 or not. The majority is towards
>adding it
>>>> in
>>>> > 2.0
>>>> > >> > (as
>>>> > >> > > it
>>>> > >> > > supports Airflow 2.0's Scalability story) and
>marking it
>>>> as
>>>> > >> > > *experimental*.
>>>> > >> > > - There were some questions raised around supporting
>this
>>>> new
>>>> > >> > > feature. So we decided that *everyone would take a
>look at
>>>> > the PR
>>>> > >> > > itself and we will spend a few minutes in the next
>>>> meeting to
>>>> > >> > decide
>>>> > >> > > whether it is 2.0 or not*.
>>>> > >> > > - *Simplification of KubernetesExecutor /
>>>> KubernetesPodOperator*
>>>> > >> > > - PR: https://github.com/apache/airflow/pull/10393
>>>> > >> > > - This will be part of *Airflow 2.0*
>>>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)*
>command *
>>>> > >> > > - WIP PR: PR:
>https://github.com/apache/airflow/pull/9467
>>>> |
>>>> > Design
>>>> > >> > > Doc:
>>>> > >> > >
>>>> > >> > >
>>>> > >> >
>>>> >
>>>>
>https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
>>>> > >> > > - *Scope*:
>>>> > >> > > - Users bash script won’t be included but
>anything in
>>>> the
>>>> > core
>>>> > >> > > Airflow would be covered
>>>> > >> > > -
>>>> > >> > >
>>>> > >> > > *DAG Definitions*:
>>>> > >> > > - Changes in Path for contrib to Providers
>packages
>>>> > >> > > - DAG Interfaces: changes in arguments of a
>DAG /
>>>> > >> > BaseOperator
>>>> > >> > > - *Configurations*:
>>>> > >> > > - Option to auto-replace deprecated configs
>with new
>>>> > options
>>>> > >> > > - *Run-time Core items*:
>>>> > >> > > - Changes like "Connection type can't be
>null". The
>>>> > >> > > upgrade-check should at least shown warning if
>it
>>>> can't
>>>> > >> > > provide option to
>>>> > >> > > detect the type.
>>>> > >> > > - *CLI refactor is out-of-scope*
>>>> > >> > > - Automatic refactor is *out-of-scope* as it
>is too
>>>> > difficult
>>>> > >> > > to cover all the cases in the Users bash
>scripts.
>>>> > >> > > - This will be covered by docs or by showing
>>>> warnings
>>>> > via the
>>>> > >> > > upgrade-check command
>>>> > >> > > - *Experimental API to New API refactor is
>>>> out-of-scope*
>>>> > (will
>>>> > >> > be
>>>> > >> > > covered by Migration docs)
>>>> > >> > > - We agreed that the airflow upgrade-check command
>*needs
>>>> to
>>>> > be
>>>> > >> > > available in the last release before Airflow 2.0*
>(1.10.x
>>>> or
>>>> > >> > 1.11.x)
>>>> > >> > > - Potential problems with time-consuming DB Migration
>were
>>>> also
>>>> > >> > > discussed. If we identify such a DB Migration (example
>the
>>>> one
>>>> > >> > involving
>>>> > >> > > TaskInstance table) should be noted separately in
>>>> Updating.md to
>>>> > >> > > provide a
>>>> > >> > > warning to the users.
>>>> > >> > > - *DEV Calls Feedback*
>>>> > >> > > - We agreed on having *Weekly calls from 7 September
>>>> onwards*
>>>> > >> > > - Calls will start with a 5-min reviewing the
>progress
>>>> from
>>>> > the
>>>> > >> > last
>>>> > >> > > call towards 2.0
>>>> > >> > > - *Process*
>>>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep
>2020
>>>> > >> > > - Changelog:
>>>> > >> > > - The current way of Changelog is OK. We don't
>need
>>>> further
>>>> > >> > > categorization like Webserver, Scheduler etc.
>>>> > >> > > - Separate Changelog would be created for
>Providers
>>>> > Packages
>>>> > >> > > - We need to figure a way to tag/label PRs &
>Issues
>>>> with
>>>> > correct
>>>> > >> > > categories. Some options that were discussed
>were:
>>>> > >> > > - Adding labels on the PRs & Issues via Bot
>>>> > >> > > - A field in PR template for PR authors to
>add, the
>>>> bot
>>>> > would
>>>> > >> > > then read the field which would be used to
>label
>>>> the PR
>>>> > >> > > - Add rules, for example Committers needs to
>add
>>>> > appropriate
>>>> > >> > > labels to the PR before merging it. We could
>have a
>>>> > >> > > scheduled Github
>>>> > >> > > Actions workflow that would fail if it finds
>PRs
>>>> without
>>>> > >> > > labels.
>>>> > >> > >
>>>> > >> > > *Things to Discuss Next*
>>>> > >> > >
>>>> > >> > > - *7 September*
>>>> > >> > > - Progress, Current Work & Discussions
>>>> > >> > > - API
>>>> > >> > > - Providers Packages
>>>> > >> > > - Discuss open questions
>>>> > >> > > - Improvements to SubDags / Concept of TaskGroup
>>>> > >> > > - AIP-34 <
>>>> https://github.com/apache/airflow/pull/10153>
>>>> > >> > > - *14 September*
>>>> > >> > > - Process:
>>>> > >> > > - When should we defer the in-scope items to
>post-2.0
>>>> > >> > > - Completion by a date?
>>>> > >> > > - Progress by a date?
>>>> > >> > > - Progress, Current Work & Discussions
>>>> > >> > > - Scheduler HA
>>>> > >> > > - Docs Improvements
>>>> > >> > > - Helm Chart
>>>> > >> > > - Discuss the issue with sources
>>>> > >> > >
>>>> > >> > >
>>>> > >> > > Regards,
>>>> > >> > > Kaxil
>>>> > >> > >
>>>> > >> >
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > >
>>>> > > Jarek Potiuk
>>>> > > Polidea | Principal Software Engineer
>>>> > >
>>>> > > M: +48 660 796 129
>>>> > >
>>>> > >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > Jarek Potiuk
>>>> > Polidea | Principal Software Engineer
>>>> >
>>>> > M: +48 660 796 129
>>>> >
>>>>
>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
>--
>
>Jarek Potiuk
>Polidea <https://www.polidea.com/> | Principal Software Engineer
>
>M: +48 660 796 129 <+48660796129>
>[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
And we have a new addition from Kamil about the need to extend slightly
plugin mechanism to be able to cover dynamically "Connections", "Connection
Form" and "Extra Links" - those are indeed the "core -> Providers"
dependencies that we still have.
They seem to be easy to handle by making providers "plugins" and extending
the plugin mechanism a bit. Thanks Kamil for the thoughtful input!
On Fri, Sep 4, 2020 at 8:08 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Just a short reminder - for some more comments/review on the "PIP package
> model of Airflow 2.0" doc
>
> https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit#
>
> I've added one small addition - in this model we want to make sure that
> there are no dependencies of core packages on any of the providers - we do
> not run such checks yet but it's easy to add.
>
> J
>
>
> On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Cool!
>>
>> If you have comments on particular sections/paragraphs - it's easier to
>> keep track of it and respond in the doc. If you have some general
>> statements, and some summary of your thinking after the review - it's best
>> to respond to the email :)
>>
>> I am ok with both and will aggregate it eventually.
>>
>> J.
>>
>>
>> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io> wrote:
>>
>>> Jarek,
>>>
>>> Thank you, this is very helpful.
>>> I assume that you would like comments in the document itself?
>>> Or, would you like them in email?
>>>
>>> Best regards,
>>> Vikram
>>>
>>>
>>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>> > As promised during the last call I prepared the proposal on how we can
>>> > approach the package model for Airflow 2.0 - including the "Provider
>>> > Packages" approach.
>>> >
>>> > https://s.apache.org/airflow-2-0-package-model
>>> >
>>> > I would like to discuss it at our next meeting on Monday. I'd love to
>>> > hear your comments.
>>> >
>>> > J.
>>> >
>>> >
>>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com>
>>> > wrote:
>>> > >
>>> > > +1 Kevin on the call :).
>>> > >
>>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com>
>>> wrote:
>>> > >>
>>> > >> Thanks Kevin, Looking forward to see you on the next call.
>>> > >>
>>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
>>> > >>
>>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to join at
>>> > least the
>>> > >> > next meeting trying to see if we can provide more perspectives on
>>> > >> > SmartSensor and anything else we can help.
>>> > >> >
>>> > >> >
>>> > >> > Cheers,
>>> > >> > Kevin Y
>>> > >> >
>>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com>
>>> > wrote:
>>> > >> >
>>> > >> > > Hi all,
>>> > >> > >
>>> > >> > > I have created a document to summarize the discussion from our
>>> > second dev
>>> > >> > > call for Airflow 2.0.
>>> > >> > >
>>> > >> > > Thank you all who joined the call.
>>> > >> > >
>>> > >> > > *Doc Link*:
>>> > >> > >
>>> > >> > >
>>> > >> >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
>>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
>>> > <
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>> >
>>> > >> > <
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>> > >
>>> > >> > > <
>>> > >> >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>>> > >> > >
>>> > >> > >
>>> > >> > > To all those who attended, can you please double-check and add
>>> if I
>>> > have
>>> > >> > > missed anything?
>>> > >> > >
>>> > >> > > To all those who didn't join, if you disagree to anything in the
>>> > Summary
>>> > >> > > please voice your opinion.
>>> > >> > >
>>> > >> > > Including the Summary here too (might potentially break
>>> formatting):
>>> > >> > >
>>> > >> > > *Key Decisions*
>>> > >> > >
>>> > >> > > - *Smart Sensors – *in 2.0 or 2.1
>>> > >> > > - AIP-17
>>> > >> > > <
>>> > >> > >
>>> > >> >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
>>> > >> > > >
>>> > >> > > |
>>> > >> > > PR: https://github.com/apache/airflow/pull/5499
>>> > >> > > - We have not come to a conclusion yet on whether this
>>> should
>>> > be
>>> > >> > > included in 2.0 or not. The majority is towards adding it
>>> in
>>> > 2.0
>>> > >> > (as
>>> > >> > > it
>>> > >> > > supports Airflow 2.0's Scalability story) and marking it
>>> as
>>> > >> > > *experimental*.
>>> > >> > > - There were some questions raised around supporting this
>>> new
>>> > >> > > feature. So we decided that *everyone would take a look at
>>> > the PR
>>> > >> > > itself and we will spend a few minutes in the next
>>> meeting to
>>> > >> > decide
>>> > >> > > whether it is 2.0 or not*.
>>> > >> > > - *Simplification of KubernetesExecutor /
>>> KubernetesPodOperator*
>>> > >> > > - PR: https://github.com/apache/airflow/pull/10393
>>> > >> > > - This will be part of *Airflow 2.0*
>>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
>>> > >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467
>>> |
>>> > Design
>>> > >> > > Doc:
>>> > >> > >
>>> > >> > >
>>> > >> >
>>> >
>>> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
>>> > >> > > - *Scope*:
>>> > >> > > - Users bash script won’t be included but anything in
>>> the
>>> > core
>>> > >> > > Airflow would be covered
>>> > >> > > -
>>> > >> > >
>>> > >> > > *DAG Definitions*:
>>> > >> > > - Changes in Path for contrib to Providers packages
>>> > >> > > - DAG Interfaces: changes in arguments of a DAG /
>>> > >> > BaseOperator
>>> > >> > > - *Configurations*:
>>> > >> > > - Option to auto-replace deprecated configs with new
>>> > options
>>> > >> > > - *Run-time Core items*:
>>> > >> > > - Changes like "Connection type can't be null". The
>>> > >> > > upgrade-check should at least shown warning if it
>>> can't
>>> > >> > > provide option to
>>> > >> > > detect the type.
>>> > >> > > - *CLI refactor is out-of-scope*
>>> > >> > > - Automatic refactor is *out-of-scope* as it is too
>>> > difficult
>>> > >> > > to cover all the cases in the Users bash scripts.
>>> > >> > > - This will be covered by docs or by showing
>>> warnings
>>> > via the
>>> > >> > > upgrade-check command
>>> > >> > > - *Experimental API to New API refactor is
>>> out-of-scope*
>>> > (will
>>> > >> > be
>>> > >> > > covered by Migration docs)
>>> > >> > > - We agreed that the airflow upgrade-check command *needs
>>> to
>>> > be
>>> > >> > > available in the last release before Airflow 2.0* (1.10.x
>>> or
>>> > >> > 1.11.x)
>>> > >> > > - Potential problems with time-consuming DB Migration were
>>> also
>>> > >> > > discussed. If we identify such a DB Migration (example the
>>> one
>>> > >> > involving
>>> > >> > > TaskInstance table) should be noted separately in
>>> Updating.md to
>>> > >> > > provide a
>>> > >> > > warning to the users.
>>> > >> > > - *DEV Calls Feedback*
>>> > >> > > - We agreed on having *Weekly calls from 7 September
>>> onwards*
>>> > >> > > - Calls will start with a 5-min reviewing the progress
>>> from
>>> > the
>>> > >> > last
>>> > >> > > call towards 2.0
>>> > >> > > - *Process*
>>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
>>> > >> > > - Changelog:
>>> > >> > > - The current way of Changelog is OK. We don't need
>>> further
>>> > >> > > categorization like Webserver, Scheduler etc.
>>> > >> > > - Separate Changelog would be created for Providers
>>> > Packages
>>> > >> > > - We need to figure a way to tag/label PRs & Issues
>>> with
>>> > correct
>>> > >> > > categories. Some options that were discussed were:
>>> > >> > > - Adding labels on the PRs & Issues via Bot
>>> > >> > > - A field in PR template for PR authors to add, the
>>> bot
>>> > would
>>> > >> > > then read the field which would be used to label
>>> the PR
>>> > >> > > - Add rules, for example Committers needs to add
>>> > appropriate
>>> > >> > > labels to the PR before merging it. We could have a
>>> > >> > > scheduled Github
>>> > >> > > Actions workflow that would fail if it finds PRs
>>> without
>>> > >> > > labels.
>>> > >> > >
>>> > >> > > *Things to Discuss Next*
>>> > >> > >
>>> > >> > > - *7 September*
>>> > >> > > - Progress, Current Work & Discussions
>>> > >> > > - API
>>> > >> > > - Providers Packages
>>> > >> > > - Discuss open questions
>>> > >> > > - Improvements to SubDags / Concept of TaskGroup
>>> > >> > > - AIP-34 <
>>> https://github.com/apache/airflow/pull/10153>
>>> > >> > > - *14 September*
>>> > >> > > - Process:
>>> > >> > > - When should we defer the in-scope items to post-2.0
>>> > >> > > - Completion by a date?
>>> > >> > > - Progress by a date?
>>> > >> > > - Progress, Current Work & Discussions
>>> > >> > > - Scheduler HA
>>> > >> > > - Docs Improvements
>>> > >> > > - Helm Chart
>>> > >> > > - Discuss the issue with sources
>>> > >> > >
>>> > >> > >
>>> > >> > > Regards,
>>> > >> > > Kaxil
>>> > >> > >
>>> > >> >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > >
>>> > > Jarek Potiuk
>>> > > Polidea | Principal Software Engineer
>>> > >
>>> > > M: +48 660 796 129
>>> > >
>>> > >
>>> >
>>> >
>>> > --
>>> >
>>> > Jarek Potiuk
>>> > Polidea | Principal Software Engineer
>>> >
>>> > M: +48 660 796 129
>>> >
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
Just a short reminder - for some more comments/review on the "PIP package
model of Airflow 2.0" doc
https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit#
I've added one small addition - in this model we want to make sure that
there are no dependencies of core packages on any of the providers - we do
not run such checks yet but it's easy to add.
J
On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Cool!
>
> If you have comments on particular sections/paragraphs - it's easier to
> keep track of it and respond in the doc. If you have some general
> statements, and some summary of your thinking after the review - it's best
> to respond to the email :)
>
> I am ok with both and will aggregate it eventually.
>
> J.
>
>
> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io> wrote:
>
>> Jarek,
>>
>> Thank you, this is very helpful.
>> I assume that you would like comments in the document itself?
>> Or, would you like them in email?
>>
>> Best regards,
>> Vikram
>>
>>
>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> > As promised during the last call I prepared the proposal on how we can
>> > approach the package model for Airflow 2.0 - including the "Provider
>> > Packages" approach.
>> >
>> > https://s.apache.org/airflow-2-0-package-model
>> >
>> > I would like to discuss it at our next meeting on Monday. I'd love to
>> > hear your comments.
>> >
>> > J.
>> >
>> >
>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <Jarek.Potiuk@polidea.com
>> >
>> > wrote:
>> > >
>> > > +1 Kevin on the call :).
>> > >
>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com>
>> wrote:
>> > >>
>> > >> Thanks Kevin, Looking forward to see you on the next call.
>> > >>
>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
>> > >>
>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to join at
>> > least the
>> > >> > next meeting trying to see if we can provide more perspectives on
>> > >> > SmartSensor and anything else we can help.
>> > >> >
>> > >> >
>> > >> > Cheers,
>> > >> > Kevin Y
>> > >> >
>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com>
>> > wrote:
>> > >> >
>> > >> > > Hi all,
>> > >> > >
>> > >> > > I have created a document to summarize the discussion from our
>> > second dev
>> > >> > > call for Airflow 2.0.
>> > >> > >
>> > >> > > Thank you all who joined the call.
>> > >> > >
>> > >> > > *Doc Link*:
>> > >> > >
>> > >> > >
>> > >> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
>> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
>> > <
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>> >
>> > >> > <
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>> > >
>> > >> > > <
>> > >> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>> > >> > >
>> > >> > >
>> > >> > > To all those who attended, can you please double-check and add
>> if I
>> > have
>> > >> > > missed anything?
>> > >> > >
>> > >> > > To all those who didn't join, if you disagree to anything in the
>> > Summary
>> > >> > > please voice your opinion.
>> > >> > >
>> > >> > > Including the Summary here too (might potentially break
>> formatting):
>> > >> > >
>> > >> > > *Key Decisions*
>> > >> > >
>> > >> > > - *Smart Sensors – *in 2.0 or 2.1
>> > >> > > - AIP-17
>> > >> > > <
>> > >> > >
>> > >> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
>> > >> > > >
>> > >> > > |
>> > >> > > PR: https://github.com/apache/airflow/pull/5499
>> > >> > > - We have not come to a conclusion yet on whether this
>> should
>> > be
>> > >> > > included in 2.0 or not. The majority is towards adding it
>> in
>> > 2.0
>> > >> > (as
>> > >> > > it
>> > >> > > supports Airflow 2.0's Scalability story) and marking it as
>> > >> > > *experimental*.
>> > >> > > - There were some questions raised around supporting this
>> new
>> > >> > > feature. So we decided that *everyone would take a look at
>> > the PR
>> > >> > > itself and we will spend a few minutes in the next meeting
>> to
>> > >> > decide
>> > >> > > whether it is 2.0 or not*.
>> > >> > > - *Simplification of KubernetesExecutor /
>> KubernetesPodOperator*
>> > >> > > - PR: https://github.com/apache/airflow/pull/10393
>> > >> > > - This will be part of *Airflow 2.0*
>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
>> > >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467
>> |
>> > Design
>> > >> > > Doc:
>> > >> > >
>> > >> > >
>> > >> >
>> >
>> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
>> > >> > > - *Scope*:
>> > >> > > - Users bash script won’t be included but anything in
>> the
>> > core
>> > >> > > Airflow would be covered
>> > >> > > -
>> > >> > >
>> > >> > > *DAG Definitions*:
>> > >> > > - Changes in Path for contrib to Providers packages
>> > >> > > - DAG Interfaces: changes in arguments of a DAG /
>> > >> > BaseOperator
>> > >> > > - *Configurations*:
>> > >> > > - Option to auto-replace deprecated configs with new
>> > options
>> > >> > > - *Run-time Core items*:
>> > >> > > - Changes like "Connection type can't be null". The
>> > >> > > upgrade-check should at least shown warning if it
>> can't
>> > >> > > provide option to
>> > >> > > detect the type.
>> > >> > > - *CLI refactor is out-of-scope*
>> > >> > > - Automatic refactor is *out-of-scope* as it is too
>> > difficult
>> > >> > > to cover all the cases in the Users bash scripts.
>> > >> > > - This will be covered by docs or by showing warnings
>> > via the
>> > >> > > upgrade-check command
>> > >> > > - *Experimental API to New API refactor is out-of-scope*
>> > (will
>> > >> > be
>> > >> > > covered by Migration docs)
>> > >> > > - We agreed that the airflow upgrade-check command *needs
>> to
>> > be
>> > >> > > available in the last release before Airflow 2.0* (1.10.x
>> or
>> > >> > 1.11.x)
>> > >> > > - Potential problems with time-consuming DB Migration were
>> also
>> > >> > > discussed. If we identify such a DB Migration (example the one
>> > >> > involving
>> > >> > > TaskInstance table) should be noted separately in Updating.md
>> to
>> > >> > > provide a
>> > >> > > warning to the users.
>> > >> > > - *DEV Calls Feedback*
>> > >> > > - We agreed on having *Weekly calls from 7 September
>> onwards*
>> > >> > > - Calls will start with a 5-min reviewing the progress from
>> > the
>> > >> > last
>> > >> > > call towards 2.0
>> > >> > > - *Process*
>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
>> > >> > > - Changelog:
>> > >> > > - The current way of Changelog is OK. We don't need
>> further
>> > >> > > categorization like Webserver, Scheduler etc.
>> > >> > > - Separate Changelog would be created for Providers
>> > Packages
>> > >> > > - We need to figure a way to tag/label PRs & Issues with
>> > correct
>> > >> > > categories. Some options that were discussed were:
>> > >> > > - Adding labels on the PRs & Issues via Bot
>> > >> > > - A field in PR template for PR authors to add, the
>> bot
>> > would
>> > >> > > then read the field which would be used to label the
>> PR
>> > >> > > - Add rules, for example Committers needs to add
>> > appropriate
>> > >> > > labels to the PR before merging it. We could have a
>> > >> > > scheduled Github
>> > >> > > Actions workflow that would fail if it finds PRs
>> without
>> > >> > > labels.
>> > >> > >
>> > >> > > *Things to Discuss Next*
>> > >> > >
>> > >> > > - *7 September*
>> > >> > > - Progress, Current Work & Discussions
>> > >> > > - API
>> > >> > > - Providers Packages
>> > >> > > - Discuss open questions
>> > >> > > - Improvements to SubDags / Concept of TaskGroup
>> > >> > > - AIP-34 <
>> https://github.com/apache/airflow/pull/10153>
>> > >> > > - *14 September*
>> > >> > > - Process:
>> > >> > > - When should we defer the in-scope items to post-2.0
>> > >> > > - Completion by a date?
>> > >> > > - Progress by a date?
>> > >> > > - Progress, Current Work & Discussions
>> > >> > > - Scheduler HA
>> > >> > > - Docs Improvements
>> > >> > > - Helm Chart
>> > >> > > - Discuss the issue with sources
>> > >> > >
>> > >> > >
>> > >> > > Regards,
>> > >> > > Kaxil
>> > >> > >
>> > >> >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Jarek Potiuk
>> > > Polidea | Principal Software Engineer
>> > >
>> > > M: +48 660 796 129
>> > >
>> > >
>> >
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea | Principal Software Engineer
>> >
>> > M: +48 660 796 129
>> >
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
Cool!
If you have comments on particular sections/paragraphs - it's easier to
keep track of it and respond in the doc. If you have some general
statements, and some summary of your thinking after the review - it's best
to respond to the email :)
I am ok with both and will aggregate it eventually.
J.
On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vi...@astronomer.io> wrote:
> Jarek,
>
> Thank you, this is very helpful.
> I assume that you would like comments in the document itself?
> Or, would you like them in email?
>
> Best regards,
> Vikram
>
>
> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > As promised during the last call I prepared the proposal on how we can
> > approach the package model for Airflow 2.0 - including the "Provider
> > Packages" approach.
> >
> > https://s.apache.org/airflow-2-0-package-model
> >
> > I would like to discuss it at our next meeting on Monday. I'd love to
> > hear your comments.
> >
> > J.
> >
> >
> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > >
> > > +1 Kevin on the call :).
> > >
> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >>
> > >> Thanks Kevin, Looking forward to see you on the next call.
> > >>
> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
> > >>
> > >> > Thank you Kaxil, this is extremely helpful. We'll try to join at
> > least the
> > >> > next meeting trying to see if we can provide more perspectives on
> > >> > SmartSensor and anything else we can help.
> > >> >
> > >> >
> > >> > Cheers,
> > >> > Kevin Y
> > >> >
> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > >> >
> > >> > > Hi all,
> > >> > >
> > >> > > I have created a document to summarize the discussion from our
> > second dev
> > >> > > call for Airflow 2.0.
> > >> > >
> > >> > > Thank you all who joined the call.
> > >> > >
> > >> > > *Doc Link*:
> > >> > >
> > >> > >
> > >> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >
> > >> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> > >
> > >> > > <
> > >> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> > >> > >
> > >> > >
> > >> > > To all those who attended, can you please double-check and add if
> I
> > have
> > >> > > missed anything?
> > >> > >
> > >> > > To all those who didn't join, if you disagree to anything in the
> > Summary
> > >> > > please voice your opinion.
> > >> > >
> > >> > > Including the Summary here too (might potentially break
> formatting):
> > >> > >
> > >> > > *Key Decisions*
> > >> > >
> > >> > > - *Smart Sensors – *in 2.0 or 2.1
> > >> > > - AIP-17
> > >> > > <
> > >> > >
> > >> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
> > >> > > >
> > >> > > |
> > >> > > PR: https://github.com/apache/airflow/pull/5499
> > >> > > - We have not come to a conclusion yet on whether this
> should
> > be
> > >> > > included in 2.0 or not. The majority is towards adding it in
> > 2.0
> > >> > (as
> > >> > > it
> > >> > > supports Airflow 2.0's Scalability story) and marking it as
> > >> > > *experimental*.
> > >> > > - There were some questions raised around supporting this
> new
> > >> > > feature. So we decided that *everyone would take a look at
> > the PR
> > >> > > itself and we will spend a few minutes in the next meeting
> to
> > >> > decide
> > >> > > whether it is 2.0 or not*.
> > >> > > - *Simplification of KubernetesExecutor /
> KubernetesPodOperator*
> > >> > > - PR: https://github.com/apache/airflow/pull/10393
> > >> > > - This will be part of *Airflow 2.0*
> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
> > >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 |
> > Design
> > >> > > Doc:
> > >> > >
> > >> > >
> > >> >
> >
> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
> > >> > > - *Scope*:
> > >> > > - Users bash script won’t be included but anything in the
> > core
> > >> > > Airflow would be covered
> > >> > > -
> > >> > >
> > >> > > *DAG Definitions*:
> > >> > > - Changes in Path for contrib to Providers packages
> > >> > > - DAG Interfaces: changes in arguments of a DAG /
> > >> > BaseOperator
> > >> > > - *Configurations*:
> > >> > > - Option to auto-replace deprecated configs with new
> > options
> > >> > > - *Run-time Core items*:
> > >> > > - Changes like "Connection type can't be null". The
> > >> > > upgrade-check should at least shown warning if it
> can't
> > >> > > provide option to
> > >> > > detect the type.
> > >> > > - *CLI refactor is out-of-scope*
> > >> > > - Automatic refactor is *out-of-scope* as it is too
> > difficult
> > >> > > to cover all the cases in the Users bash scripts.
> > >> > > - This will be covered by docs or by showing warnings
> > via the
> > >> > > upgrade-check command
> > >> > > - *Experimental API to New API refactor is out-of-scope*
> > (will
> > >> > be
> > >> > > covered by Migration docs)
> > >> > > - We agreed that the airflow upgrade-check command *needs to
> > be
> > >> > > available in the last release before Airflow 2.0* (1.10.x or
> > >> > 1.11.x)
> > >> > > - Potential problems with time-consuming DB Migration were also
> > >> > > discussed. If we identify such a DB Migration (example the one
> > >> > involving
> > >> > > TaskInstance table) should be noted separately in Updating.md
> to
> > >> > > provide a
> > >> > > warning to the users.
> > >> > > - *DEV Calls Feedback*
> > >> > > - We agreed on having *Weekly calls from 7 September
> onwards*
> > >> > > - Calls will start with a 5-min reviewing the progress from
> > the
> > >> > last
> > >> > > call towards 2.0
> > >> > > - *Process*
> > >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
> > >> > > - Changelog:
> > >> > > - The current way of Changelog is OK. We don't need
> further
> > >> > > categorization like Webserver, Scheduler etc.
> > >> > > - Separate Changelog would be created for Providers
> > Packages
> > >> > > - We need to figure a way to tag/label PRs & Issues with
> > correct
> > >> > > categories. Some options that were discussed were:
> > >> > > - Adding labels on the PRs & Issues via Bot
> > >> > > - A field in PR template for PR authors to add, the
> bot
> > would
> > >> > > then read the field which would be used to label the
> PR
> > >> > > - Add rules, for example Committers needs to add
> > appropriate
> > >> > > labels to the PR before merging it. We could have a
> > >> > > scheduled Github
> > >> > > Actions workflow that would fail if it finds PRs
> without
> > >> > > labels.
> > >> > >
> > >> > > *Things to Discuss Next*
> > >> > >
> > >> > > - *7 September*
> > >> > > - Progress, Current Work & Discussions
> > >> > > - API
> > >> > > - Providers Packages
> > >> > > - Discuss open questions
> > >> > > - Improvements to SubDags / Concept of TaskGroup
> > >> > > - AIP-34 <
> https://github.com/apache/airflow/pull/10153>
> > >> > > - *14 September*
> > >> > > - Process:
> > >> > > - When should we defer the in-scope items to post-2.0
> > >> > > - Completion by a date?
> > >> > > - Progress by a date?
> > >> > > - Progress, Current Work & Discussions
> > >> > > - Scheduler HA
> > >> > > - Docs Improvements
> > >> > > - Helm Chart
> > >> > > - Discuss the issue with sources
> > >> > >
> > >> > >
> > >> > > Regards,
> > >> > > Kaxil
> > >> > >
> > >> >
> > >
> > >
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea | Principal Software Engineer
> > >
> > > M: +48 660 796 129
> > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Vikram Koka <vi...@astronomer.io>.
Jarek,
Thank you, this is very helpful.
I assume that you would like comments in the document itself?
Or, would you like them in email?
Best regards,
Vikram
On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> As promised during the last call I prepared the proposal on how we can
> approach the package model for Airflow 2.0 - including the "Provider
> Packages" approach.
>
> https://s.apache.org/airflow-2-0-package-model
>
> I would like to discuss it at our next meeting on Monday. I'd love to
> hear your comments.
>
> J.
>
>
> On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
> >
> > +1 Kevin on the call :).
> >
> > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com> wrote:
> >>
> >> Thanks Kevin, Looking forward to see you on the next call.
> >>
> >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
> >>
> >> > Thank you Kaxil, this is extremely helpful. We'll try to join at
> least the
> >> > next meeting trying to see if we can provide more perspectives on
> >> > SmartSensor and anything else we can help.
> >> >
> >> >
> >> > Cheers,
> >> > Kevin Y
> >> >
> >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com>
> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I have created a document to summarize the discussion from our
> second dev
> >> > > call for Airflow 2.0.
> >> > >
> >> > > Thank you all who joined the call.
> >> > >
> >> > > *Doc Link*:
> >> > >
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
> >> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >
> >> > > <
> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >> > >
> >> > >
> >> > > To all those who attended, can you please double-check and add if I
> have
> >> > > missed anything?
> >> > >
> >> > > To all those who didn't join, if you disagree to anything in the
> Summary
> >> > > please voice your opinion.
> >> > >
> >> > > Including the Summary here too (might potentially break formatting):
> >> > >
> >> > > *Key Decisions*
> >> > >
> >> > > - *Smart Sensors – *in 2.0 or 2.1
> >> > > - AIP-17
> >> > > <
> >> > >
> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
> >> > > >
> >> > > |
> >> > > PR: https://github.com/apache/airflow/pull/5499
> >> > > - We have not come to a conclusion yet on whether this should
> be
> >> > > included in 2.0 or not. The majority is towards adding it in
> 2.0
> >> > (as
> >> > > it
> >> > > supports Airflow 2.0's Scalability story) and marking it as
> >> > > *experimental*.
> >> > > - There were some questions raised around supporting this new
> >> > > feature. So we decided that *everyone would take a look at
> the PR
> >> > > itself and we will spend a few minutes in the next meeting to
> >> > decide
> >> > > whether it is 2.0 or not*.
> >> > > - *Simplification of KubernetesExecutor / KubernetesPodOperator*
> >> > > - PR: https://github.com/apache/airflow/pull/10393
> >> > > - This will be part of *Airflow 2.0*
> >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
> >> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 |
> Design
> >> > > Doc:
> >> > >
> >> > >
> >> >
> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
> >> > > - *Scope*:
> >> > > - Users bash script won’t be included but anything in the
> core
> >> > > Airflow would be covered
> >> > > -
> >> > >
> >> > > *DAG Definitions*:
> >> > > - Changes in Path for contrib to Providers packages
> >> > > - DAG Interfaces: changes in arguments of a DAG /
> >> > BaseOperator
> >> > > - *Configurations*:
> >> > > - Option to auto-replace deprecated configs with new
> options
> >> > > - *Run-time Core items*:
> >> > > - Changes like "Connection type can't be null". The
> >> > > upgrade-check should at least shown warning if it can't
> >> > > provide option to
> >> > > detect the type.
> >> > > - *CLI refactor is out-of-scope*
> >> > > - Automatic refactor is *out-of-scope* as it is too
> difficult
> >> > > to cover all the cases in the Users bash scripts.
> >> > > - This will be covered by docs or by showing warnings
> via the
> >> > > upgrade-check command
> >> > > - *Experimental API to New API refactor is out-of-scope*
> (will
> >> > be
> >> > > covered by Migration docs)
> >> > > - We agreed that the airflow upgrade-check command *needs to
> be
> >> > > available in the last release before Airflow 2.0* (1.10.x or
> >> > 1.11.x)
> >> > > - Potential problems with time-consuming DB Migration were also
> >> > > discussed. If we identify such a DB Migration (example the one
> >> > involving
> >> > > TaskInstance table) should be noted separately in Updating.md to
> >> > > provide a
> >> > > warning to the users.
> >> > > - *DEV Calls Feedback*
> >> > > - We agreed on having *Weekly calls from 7 September onwards*
> >> > > - Calls will start with a 5-min reviewing the progress from
> the
> >> > last
> >> > > call towards 2.0
> >> > > - *Process*
> >> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
> >> > > - Changelog:
> >> > > - The current way of Changelog is OK. We don't need further
> >> > > categorization like Webserver, Scheduler etc.
> >> > > - Separate Changelog would be created for Providers
> Packages
> >> > > - We need to figure a way to tag/label PRs & Issues with
> correct
> >> > > categories. Some options that were discussed were:
> >> > > - Adding labels on the PRs & Issues via Bot
> >> > > - A field in PR template for PR authors to add, the bot
> would
> >> > > then read the field which would be used to label the PR
> >> > > - Add rules, for example Committers needs to add
> appropriate
> >> > > labels to the PR before merging it. We could have a
> >> > > scheduled Github
> >> > > Actions workflow that would fail if it finds PRs without
> >> > > labels.
> >> > >
> >> > > *Things to Discuss Next*
> >> > >
> >> > > - *7 September*
> >> > > - Progress, Current Work & Discussions
> >> > > - API
> >> > > - Providers Packages
> >> > > - Discuss open questions
> >> > > - Improvements to SubDags / Concept of TaskGroup
> >> > > - AIP-34 <https://github.com/apache/airflow/pull/10153>
> >> > > - *14 September*
> >> > > - Process:
> >> > > - When should we defer the in-scope items to post-2.0
> >> > > - Completion by a date?
> >> > > - Progress by a date?
> >> > > - Progress, Current Work & Discussions
> >> > > - Scheduler HA
> >> > > - Docs Improvements
> >> > > - Helm Chart
> >> > > - Discuss the issue with sources
> >> > >
> >> > >
> >> > > Regards,
> >> > > Kaxil
> >> > >
> >> >
> >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
> >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
As promised during the last call I prepared the proposal on how we can
approach the package model for Airflow 2.0 - including the "Provider
Packages" approach.
https://s.apache.org/airflow-2-0-package-model
I would like to discuss it at our next meeting on Monday. I'd love to
hear your comments.
J.
On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk <Ja...@polidea.com> wrote:
>
> +1 Kevin on the call :).
>
> On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com> wrote:
>>
>> Thanks Kevin, Looking forward to see you on the next call.
>>
>> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
>>
>> > Thank you Kaxil, this is extremely helpful. We'll try to join at least the
>> > next meeting trying to see if we can provide more perspectives on
>> > SmartSensor and anything else we can help.
>> >
>> >
>> > Cheers,
>> > Kevin Y
>> >
>> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com> wrote:
>> >
>> > > Hi all,
>> > >
>> > > I have created a document to summarize the discussion from our second dev
>> > > call for Airflow 2.0.
>> > >
>> > > Thank you all who joined the call.
>> > >
>> > > *Doc Link*:
>> > >
>> > >
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
>> > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
>> > > <
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
>> > >
>> > >
>> > > To all those who attended, can you please double-check and add if I have
>> > > missed anything?
>> > >
>> > > To all those who didn't join, if you disagree to anything in the Summary
>> > > please voice your opinion.
>> > >
>> > > Including the Summary here too (might potentially break formatting):
>> > >
>> > > *Key Decisions*
>> > >
>> > > - *Smart Sensors – *in 2.0 or 2.1
>> > > - AIP-17
>> > > <
>> > >
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
>> > > >
>> > > |
>> > > PR: https://github.com/apache/airflow/pull/5499
>> > > - We have not come to a conclusion yet on whether this should be
>> > > included in 2.0 or not. The majority is towards adding it in 2.0
>> > (as
>> > > it
>> > > supports Airflow 2.0's Scalability story) and marking it as
>> > > *experimental*.
>> > > - There were some questions raised around supporting this new
>> > > feature. So we decided that *everyone would take a look at the PR
>> > > itself and we will spend a few minutes in the next meeting to
>> > decide
>> > > whether it is 2.0 or not*.
>> > > - *Simplification of KubernetesExecutor / KubernetesPodOperator*
>> > > - PR: https://github.com/apache/airflow/pull/10393
>> > > - This will be part of *Airflow 2.0*
>> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
>> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 | Design
>> > > Doc:
>> > >
>> > >
>> > https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
>> > > - *Scope*:
>> > > - Users bash script won’t be included but anything in the core
>> > > Airflow would be covered
>> > > -
>> > >
>> > > *DAG Definitions*:
>> > > - Changes in Path for contrib to Providers packages
>> > > - DAG Interfaces: changes in arguments of a DAG /
>> > BaseOperator
>> > > - *Configurations*:
>> > > - Option to auto-replace deprecated configs with new options
>> > > - *Run-time Core items*:
>> > > - Changes like "Connection type can't be null". The
>> > > upgrade-check should at least shown warning if it can't
>> > > provide option to
>> > > detect the type.
>> > > - *CLI refactor is out-of-scope*
>> > > - Automatic refactor is *out-of-scope* as it is too difficult
>> > > to cover all the cases in the Users bash scripts.
>> > > - This will be covered by docs or by showing warnings via the
>> > > upgrade-check command
>> > > - *Experimental API to New API refactor is out-of-scope* (will
>> > be
>> > > covered by Migration docs)
>> > > - We agreed that the airflow upgrade-check command *needs to be
>> > > available in the last release before Airflow 2.0* (1.10.x or
>> > 1.11.x)
>> > > - Potential problems with time-consuming DB Migration were also
>> > > discussed. If we identify such a DB Migration (example the one
>> > involving
>> > > TaskInstance table) should be noted separately in Updating.md to
>> > > provide a
>> > > warning to the users.
>> > > - *DEV Calls Feedback*
>> > > - We agreed on having *Weekly calls from 7 September onwards*
>> > > - Calls will start with a 5-min reviewing the progress from the
>> > last
>> > > call towards 2.0
>> > > - *Process*
>> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
>> > > - Changelog:
>> > > - The current way of Changelog is OK. We don't need further
>> > > categorization like Webserver, Scheduler etc.
>> > > - Separate Changelog would be created for Providers Packages
>> > > - We need to figure a way to tag/label PRs & Issues with correct
>> > > categories. Some options that were discussed were:
>> > > - Adding labels on the PRs & Issues via Bot
>> > > - A field in PR template for PR authors to add, the bot would
>> > > then read the field which would be used to label the PR
>> > > - Add rules, for example Committers needs to add appropriate
>> > > labels to the PR before merging it. We could have a
>> > > scheduled Github
>> > > Actions workflow that would fail if it finds PRs without
>> > > labels.
>> > >
>> > > *Things to Discuss Next*
>> > >
>> > > - *7 September*
>> > > - Progress, Current Work & Discussions
>> > > - API
>> > > - Providers Packages
>> > > - Discuss open questions
>> > > - Improvements to SubDags / Concept of TaskGroup
>> > > - AIP-34 <https://github.com/apache/airflow/pull/10153>
>> > > - *14 September*
>> > > - Process:
>> > > - When should we defer the in-scope items to post-2.0
>> > > - Completion by a date?
>> > > - Progress by a date?
>> > > - Progress, Current Work & Discussions
>> > > - Scheduler HA
>> > > - Docs Improvements
>> > > - Helm Chart
>> > > - Discuss the issue with sources
>> > >
>> > >
>> > > Regards,
>> > > Kaxil
>> > >
>> >
>
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>
>
--
Jarek Potiuk
Polidea | Principal Software Engineer
M: +48 660 796 129
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
+1 Kevin on the call :).
On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik <ka...@gmail.com> wrote:
> Thanks Kevin, Looking forward to see you on the next call.
>
> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
>
> > Thank you Kaxil, this is extremely helpful. We'll try to join at least
> the
> > next meeting trying to see if we can provide more perspectives on
> > SmartSensor and anything else we can help.
> >
> >
> > Cheers,
> > Kevin Y
> >
> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I have created a document to summarize the discussion from our second
> dev
> > > call for Airflow 2.0.
> > >
> > > Thank you all who joined the call.
> > >
> > > *Doc Link*:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> > >
> > >
> > > To all those who attended, can you please double-check and add if I
> have
> > > missed anything?
> > >
> > > To all those who didn't join, if you disagree to anything in the
> Summary
> > > please voice your opinion.
> > >
> > > Including the Summary here too (might potentially break formatting):
> > >
> > > *Key Decisions*
> > >
> > > - *Smart Sensors – *in 2.0 or 2.1
> > > - AIP-17
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
> > > >
> > > |
> > > PR: https://github.com/apache/airflow/pull/5499
> > > - We have not come to a conclusion yet on whether this should be
> > > included in 2.0 or not. The majority is towards adding it in 2.0
> > (as
> > > it
> > > supports Airflow 2.0's Scalability story) and marking it as
> > > *experimental*.
> > > - There were some questions raised around supporting this new
> > > feature. So we decided that *everyone would take a look at the PR
> > > itself and we will spend a few minutes in the next meeting to
> > decide
> > > whether it is 2.0 or not*.
> > > - *Simplification of KubernetesExecutor / KubernetesPodOperator*
> > > - PR: https://github.com/apache/airflow/pull/10393
> > > - This will be part of *Airflow 2.0*
> > > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
> > > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 |
> Design
> > > Doc:
> > >
> > >
> >
> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
> > > - *Scope*:
> > > - Users bash script won’t be included but anything in the core
> > > Airflow would be covered
> > > -
> > >
> > > *DAG Definitions*:
> > > - Changes in Path for contrib to Providers packages
> > > - DAG Interfaces: changes in arguments of a DAG /
> > BaseOperator
> > > - *Configurations*:
> > > - Option to auto-replace deprecated configs with new
> options
> > > - *Run-time Core items*:
> > > - Changes like "Connection type can't be null". The
> > > upgrade-check should at least shown warning if it can't
> > > provide option to
> > > detect the type.
> > > - *CLI refactor is out-of-scope*
> > > - Automatic refactor is *out-of-scope* as it is too
> difficult
> > > to cover all the cases in the Users bash scripts.
> > > - This will be covered by docs or by showing warnings via
> the
> > > upgrade-check command
> > > - *Experimental API to New API refactor is out-of-scope* (will
> > be
> > > covered by Migration docs)
> > > - We agreed that the airflow upgrade-check command *needs to be
> > > available in the last release before Airflow 2.0* (1.10.x or
> > 1.11.x)
> > > - Potential problems with time-consuming DB Migration were also
> > > discussed. If we identify such a DB Migration (example the one
> > involving
> > > TaskInstance table) should be noted separately in Updating.md to
> > > provide a
> > > warning to the users.
> > > - *DEV Calls Feedback*
> > > - We agreed on having *Weekly calls from 7 September onwards*
> > > - Calls will start with a 5-min reviewing the progress from the
> > last
> > > call towards 2.0
> > > - *Process*
> > > - A *2.0.0-test* branch will be created on 10 Sep 2020
> > > - Changelog:
> > > - The current way of Changelog is OK. We don't need further
> > > categorization like Webserver, Scheduler etc.
> > > - Separate Changelog would be created for Providers Packages
> > > - We need to figure a way to tag/label PRs & Issues with
> correct
> > > categories. Some options that were discussed were:
> > > - Adding labels on the PRs & Issues via Bot
> > > - A field in PR template for PR authors to add, the bot
> would
> > > then read the field which would be used to label the PR
> > > - Add rules, for example Committers needs to add
> appropriate
> > > labels to the PR before merging it. We could have a
> > > scheduled Github
> > > Actions workflow that would fail if it finds PRs without
> > > labels.
> > >
> > > *Things to Discuss Next*
> > >
> > > - *7 September*
> > > - Progress, Current Work & Discussions
> > > - API
> > > - Providers Packages
> > > - Discuss open questions
> > > - Improvements to SubDags / Concept of TaskGroup
> > > - AIP-34 <https://github.com/apache/airflow/pull/10153>
> > > - *14 September*
> > > - Process:
> > > - When should we defer the in-scope items to post-2.0
> > > - Completion by a date?
> > > - Progress by a date?
> > > - Progress, Current Work & Discussions
> > > - Scheduler HA
> > > - Docs Improvements
> > > - Helm Chart
> > > - Discuss the issue with sources
> > >
> > >
> > > Regards,
> > > Kaxil
> > >
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks Kevin, Looking forward to see you on the next call.
On Wed, Aug 26, 2020, 08:54 Kevin Yang <yr...@gmail.com> wrote:
> Thank you Kaxil, this is extremely helpful. We'll try to join at least the
> next meeting trying to see if we can provide more perspectives on
> SmartSensor and anything else we can help.
>
>
> Cheers,
> Kevin Y
>
> On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Hi all,
> >
> > I have created a document to summarize the discussion from our second dev
> > call for Airflow 2.0.
> >
> > Thank you all who joined the call.
> >
> > *Doc Link*:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020
> >
> >
> > To all those who attended, can you please double-check and add if I have
> > missed anything?
> >
> > To all those who didn't join, if you disagree to anything in the Summary
> > please voice your opinion.
> >
> > Including the Summary here too (might potentially break formatting):
> >
> > *Key Decisions*
> >
> > - *Smart Sensors – *in 2.0 or 2.1
> > - AIP-17
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
> > >
> > |
> > PR: https://github.com/apache/airflow/pull/5499
> > - We have not come to a conclusion yet on whether this should be
> > included in 2.0 or not. The majority is towards adding it in 2.0
> (as
> > it
> > supports Airflow 2.0's Scalability story) and marking it as
> > *experimental*.
> > - There were some questions raised around supporting this new
> > feature. So we decided that *everyone would take a look at the PR
> > itself and we will spend a few minutes in the next meeting to
> decide
> > whether it is 2.0 or not*.
> > - *Simplification of KubernetesExecutor / KubernetesPodOperator*
> > - PR: https://github.com/apache/airflow/pull/10393
> > - This will be part of *Airflow 2.0*
> > - *Airflow Upgrade Check* (airflow upgrade-check)* command *
> > - WIP PR: PR: https://github.com/apache/airflow/pull/9467 | Design
> > Doc:
> >
> >
> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
> > - *Scope*:
> > - Users bash script won’t be included but anything in the core
> > Airflow would be covered
> > -
> >
> > *DAG Definitions*:
> > - Changes in Path for contrib to Providers packages
> > - DAG Interfaces: changes in arguments of a DAG /
> BaseOperator
> > - *Configurations*:
> > - Option to auto-replace deprecated configs with new options
> > - *Run-time Core items*:
> > - Changes like "Connection type can't be null". The
> > upgrade-check should at least shown warning if it can't
> > provide option to
> > detect the type.
> > - *CLI refactor is out-of-scope*
> > - Automatic refactor is *out-of-scope* as it is too difficult
> > to cover all the cases in the Users bash scripts.
> > - This will be covered by docs or by showing warnings via the
> > upgrade-check command
> > - *Experimental API to New API refactor is out-of-scope* (will
> be
> > covered by Migration docs)
> > - We agreed that the airflow upgrade-check command *needs to be
> > available in the last release before Airflow 2.0* (1.10.x or
> 1.11.x)
> > - Potential problems with time-consuming DB Migration were also
> > discussed. If we identify such a DB Migration (example the one
> involving
> > TaskInstance table) should be noted separately in Updating.md to
> > provide a
> > warning to the users.
> > - *DEV Calls Feedback*
> > - We agreed on having *Weekly calls from 7 September onwards*
> > - Calls will start with a 5-min reviewing the progress from the
> last
> > call towards 2.0
> > - *Process*
> > - A *2.0.0-test* branch will be created on 10 Sep 2020
> > - Changelog:
> > - The current way of Changelog is OK. We don't need further
> > categorization like Webserver, Scheduler etc.
> > - Separate Changelog would be created for Providers Packages
> > - We need to figure a way to tag/label PRs & Issues with correct
> > categories. Some options that were discussed were:
> > - Adding labels on the PRs & Issues via Bot
> > - A field in PR template for PR authors to add, the bot would
> > then read the field which would be used to label the PR
> > - Add rules, for example Committers needs to add appropriate
> > labels to the PR before merging it. We could have a
> > scheduled Github
> > Actions workflow that would fail if it finds PRs without
> > labels.
> >
> > *Things to Discuss Next*
> >
> > - *7 September*
> > - Progress, Current Work & Discussions
> > - API
> > - Providers Packages
> > - Discuss open questions
> > - Improvements to SubDags / Concept of TaskGroup
> > - AIP-34 <https://github.com/apache/airflow/pull/10153>
> > - *14 September*
> > - Process:
> > - When should we defer the in-scope items to post-2.0
> > - Completion by a date?
> > - Progress by a date?
> > - Progress, Current Work & Discussions
> > - Scheduler HA
> > - Docs Improvements
> > - Helm Chart
> > - Discuss the issue with sources
> >
> >
> > Regards,
> > Kaxil
> >
>
Re: [Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020
Posted by Kevin Yang <yr...@gmail.com>.
Thank you Kaxil, this is extremely helpful. We'll try to join at least the
next meeting trying to see if we can provide more perspectives on
SmartSensor and anything else we can help.
Cheers,
Kevin Y
On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik <ka...@gmail.com> wrote:
> Hi all,
>
> I have created a document to summarize the discussion from our second dev
> call for Airflow 2.0.
>
> Thank you all who joined the call.
>
> *Doc Link*:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020>
>
> To all those who attended, can you please double-check and add if I have
> missed anything?
>
> To all those who didn't join, if you disagree to anything in the Summary
> please voice your opinion.
>
> Including the Summary here too (might potentially break formatting):
>
> *Key Decisions*
>
> - *Smart Sensors – *in 2.0 or 2.1
> - AIP-17
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
> >
> |
> PR: https://github.com/apache/airflow/pull/5499
> - We have not come to a conclusion yet on whether this should be
> included in 2.0 or not. The majority is towards adding it in 2.0 (as
> it
> supports Airflow 2.0's Scalability story) and marking it as
> *experimental*.
> - There were some questions raised around supporting this new
> feature. So we decided that *everyone would take a look at the PR
> itself and we will spend a few minutes in the next meeting to decide
> whether it is 2.0 or not*.
> - *Simplification of KubernetesExecutor / KubernetesPodOperator*
> - PR: https://github.com/apache/airflow/pull/10393
> - This will be part of *Airflow 2.0*
> - *Airflow Upgrade Check* (airflow upgrade-check)* command *
> - WIP PR: PR: https://github.com/apache/airflow/pull/9467 | Design
> Doc:
>
> https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv
> - *Scope*:
> - Users bash script won’t be included but anything in the core
> Airflow would be covered
> -
>
> *DAG Definitions*:
> - Changes in Path for contrib to Providers packages
> - DAG Interfaces: changes in arguments of a DAG / BaseOperator
> - *Configurations*:
> - Option to auto-replace deprecated configs with new options
> - *Run-time Core items*:
> - Changes like "Connection type can't be null". The
> upgrade-check should at least shown warning if it can't
> provide option to
> detect the type.
> - *CLI refactor is out-of-scope*
> - Automatic refactor is *out-of-scope* as it is too difficult
> to cover all the cases in the Users bash scripts.
> - This will be covered by docs or by showing warnings via the
> upgrade-check command
> - *Experimental API to New API refactor is out-of-scope* (will be
> covered by Migration docs)
> - We agreed that the airflow upgrade-check command *needs to be
> available in the last release before Airflow 2.0* (1.10.x or 1.11.x)
> - Potential problems with time-consuming DB Migration were also
> discussed. If we identify such a DB Migration (example the one involving
> TaskInstance table) should be noted separately in Updating.md to
> provide a
> warning to the users.
> - *DEV Calls Feedback*
> - We agreed on having *Weekly calls from 7 September onwards*
> - Calls will start with a 5-min reviewing the progress from the last
> call towards 2.0
> - *Process*
> - A *2.0.0-test* branch will be created on 10 Sep 2020
> - Changelog:
> - The current way of Changelog is OK. We don't need further
> categorization like Webserver, Scheduler etc.
> - Separate Changelog would be created for Providers Packages
> - We need to figure a way to tag/label PRs & Issues with correct
> categories. Some options that were discussed were:
> - Adding labels on the PRs & Issues via Bot
> - A field in PR template for PR authors to add, the bot would
> then read the field which would be used to label the PR
> - Add rules, for example Committers needs to add appropriate
> labels to the PR before merging it. We could have a
> scheduled Github
> Actions workflow that would fail if it finds PRs without
> labels.
>
> *Things to Discuss Next*
>
> - *7 September*
> - Progress, Current Work & Discussions
> - API
> - Providers Packages
> - Discuss open questions
> - Improvements to SubDags / Concept of TaskGroup
> - AIP-34 <https://github.com/apache/airflow/pull/10153>
> - *14 September*
> - Process:
> - When should we defer the in-scope items to post-2.0
> - Completion by a date?
> - Progress by a date?
> - Progress, Current Work & Discussions
> - Scheduler HA
> - Docs Improvements
> - Helm Chart
> - Discuss the issue with sources
>
>
> Regards,
> Kaxil
>