You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Kaxil Naik <ka...@gmail.com> on 2020/09/09 10:29:00 UTC
[Meeting Notes] Airflow 2.0 Dev Call #3 - 7 Sep 2020
Hi all,
I have created a document to summarize the discussion from our third dev
call for Airflow 2.0.
Thank you all who joined the call.
*Doc Link*:
https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#3:7Sep2020
To all those who attended, can you please double-check and add if I have
missed anything?
To all those who didn't join, if you disagree to anything in the Summary please
voice your opinion.
Including the Summary here too (might potentially break formatting):
*Key Decisions*
- *Smart Sensors*
- Will be included in 2.0 as an *early-access* feature with a clear
note that *this feature might potentially change in future Airflow
version with breaking changes*.
- Airbnb team would be happy to help on the support side answering
questions related to Smart Sensor.
- Add docs around different execution modes for Sensor: *Poke
mode*, *Reschedule
mode* vs *Smart Sensor*
- *Providers Packages*
- We had a consensus on *releasing providers packages separately* mainly
because of the following reasons:
- Separate cadence for providers compared to Airflow, so bugs in
operator/hooks can be fixed lot faster.
- Enterprises generally would not like to upgrade the “core”
(Scheduler) as a small bug can break the deployment and
affect all the DAGs
- Breaking library changes (new version of a library) can be fixed
with a new version of Backport/Providers
- Upgrades of backport providers are not “that” destructive i.e.
even if you upgrade to a newer version and find a bug, you
could go back to
the previous version without causing any issues at all.
- Open questions / Action Items:
- How would users figure out “breaking changes” with CALVER
Versioning (which is very clear with SEMVER)?
- Use plugin Mechanism to:
- Register Connections from an external provider to allow
custom field or hide existing form fields.
- Register Operator Extra links
<https://airflow.apache.org/docs/stable/howto/define_extra_link.html>
for
operators in providers so that a change is not required in Airflow
- Backport providers will only be supported/released for *three
months after 2.0.0 released*
- *Timeline to Airflow 2.0*
- Only *critical fixes* (fixes to bugs that takedown Production
system) will be backported to 1.10.x core for *six months* after
Airflow 2.0 is released.
Date
Milestone
Week of 7 Sep 2020 Create the 2.0.0-test branch
While the scope is fluid, we would be rebasing this test branch from
master. After we completely freeze the scope, we would only cherrypick
commits from Airflow Master to v2-0-test branch if they are “in-scope”.
Normal development would continue on Master branch i.e. PRs would be
created against Airflow Master.
Week of 28 Sep 2020 Cut Functionally complete 2.0 alpha release
Week of 12 Oct 2020 Cut first 2.0 beta release
Beta snapshots would be published to the Airflow Community to test and
create issues to make sure Airflow is functioning and backwards compatible.
Week of 19 Oct 2020 Cut bridge release based on 1.10.x - jump-off point to
2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0
Week of 26 Oct 2020 Cut second 2.0 beta release
Week of 9 Nov 2020 Cut third 2.0 beta release
Week of 23 Nov 2020 Cut first 2.0 release candidate (2.0.0rc1)
*Things to Discuss Next*
- *14 September (Subject to change)*
- API
- Progress, Current Work & Discussions
- Any open questions?
- Improvements to SubDags / Concept of TaskGroup
- *AIP-34*:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator
- *PR*: https://github.com/apache/airflow/pull/10153
- Any concerns on including this PR to 2.0 ??
- Do we want to support both TaskGroup & SubDags?
- Process:
- When should we defer the in-scope items to post-2.0
- Completion by a date?
- Progress by a date?
- Progress, Current Work & Discussions
- Scheduler HA
- Docs Improvements
- Helm Chart
- Discuss the issue with sources
Regards,
Kaxil
Re: [Meeting Notes] Airflow 2.0 Dev Call #3 - 7 Sep 2020
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks, Vikram for updating the planning page.
On Wed, Sep 9, 2020 at 5:43 PM Vikram Koka <vi...@astronomer.io> wrote:
> Thanks Kaxil, this looks right to me as well.
> I updated the main Airflow 2.0 planning page as well to reflect the current
> scope and milestones based on this meeting.
>
> *Doc Link*:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning
>
> I also wanted to thank Kevin from the AirBnb team for joining the call on
> Monday and offering to help with SmartSensors support as needed. I am
> excited that this feature will be part of the 2.0 release.
>
> Vikram
>
>
> On Wed, Sep 9, 2020 at 3:41 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > LGTM! Thanks, Kaxil for putting this together. It is really helpful!
> >
> > On Wed, Sep 9, 2020 at 12:29 PM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I have created a document to summarize the discussion from our third
> dev
> > > call for Airflow 2.0.
> > >
> > > Thank you all who joined the call.
> > >
> > > *Doc Link*:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#3:7Sep2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020
> >
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020
> > >
> > >
> > > To all those who attended, can you please double-check and add if I
> have
> > > missed anything?
> > >
> > > To all those who didn't join, if you disagree to anything in the
> Summary
> > > please
> > > voice your opinion.
> > >
> > > Including the Summary here too (might potentially break formatting):
> > >
> > > *Key Decisions*
> > >
> > > - *Smart Sensors*
> > > - Will be included in 2.0 as an *early-access* feature with a
> clear
> > > note that *this feature might potentially change in future
> Airflow
> > > version with breaking changes*.
> > > - Airbnb team would be happy to help on the support side
> answering
> > > questions related to Smart Sensor.
> > > - Add docs around different execution modes for Sensor: *Poke
> > > mode*, *Reschedule
> > > mode* vs *Smart Sensor*
> > > - *Providers Packages*
> > > - We had a consensus on *releasing providers packages separately*
> > > mainly
> > > because of the following reasons:
> > > - Separate cadence for providers compared to Airflow, so bugs
> in
> > > operator/hooks can be fixed lot faster.
> > > - Enterprises generally would not like to upgrade the “core”
> > > (Scheduler) as a small bug can break the deployment and
> > > affect all the DAGs
> > > - Breaking library changes (new version of a library) can be
> > fixed
> > > with a new version of Backport/Providers
> > > - Upgrades of backport providers are not “that” destructive
> i.e.
> > > even if you upgrade to a newer version and find a bug, you
> > > could go back to
> > > the previous version without causing any issues at all.
> > > - Open questions / Action Items:
> > > - How would users figure out “breaking changes” with CALVER
> > > Versioning (which is very clear with SEMVER)?
> > > - Use plugin Mechanism to:
> > > - Register Connections from an external provider to allow
> > > custom field or hide existing form fields.
> > > - Register Operator Extra links
> > > <
> > > https://airflow.apache.org/docs/stable/howto/define_extra_link.html>
> > > for
> > > operators in providers so that a change is not required in
> > > Airflow
> > > - Backport providers will only be supported/released for
> *three
> > > months after 2.0.0 released*
> > > - *Timeline to Airflow 2.0*
> > > - Only *critical fixes* (fixes to bugs that takedown Production
> > > system) will be backported to 1.10.x core for *six months* after
> > > Airflow 2.0 is released.
> > >
> > > Date
> > >
> > > Milestone
> > >
> > > Week of 7 Sep 2020 Create the 2.0.0-test branch
> > >
> > > While the scope is fluid, we would be rebasing this test branch from
> > > master. After we completely freeze the scope, we would only cherrypick
> > > commits from Airflow Master to v2-0-test branch if they are “in-scope”.
> > > Normal development would continue on Master branch i.e. PRs would be
> > > created against Airflow Master.
> > > Week of 28 Sep 2020 Cut Functionally complete 2.0 alpha release
> > > Week of 12 Oct 2020 Cut first 2.0 beta release
> > >
> > > Beta snapshots would be published to the Airflow Community to test and
> > > create issues to make sure Airflow is functioning and backwards
> > compatible.
> > > Week of 19 Oct 2020 Cut bridge release based on 1.10.x - jump-off point
> > to
> > > 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for
> 2.0
> > > Week of 26 Oct 2020 Cut second 2.0 beta release
> > > Week of 9 Nov 2020 Cut third 2.0 beta release
> > > Week of 23 Nov 2020 Cut first 2.0 release candidate (2.0.0rc1)
> > >
> > > *Things to Discuss Next*
> > >
> > > - *14 September (Subject to change)*
> > > - API
> > > - Progress, Current Work & Discussions
> > > - Any open questions?
> > > - Improvements to SubDags / Concept of TaskGroup
> > > - *AIP-34*:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator
> > >
> > > - *PR*: https://github.com/apache/airflow/pull/10153
> > > - Any concerns on including this PR to 2.0 ??
> > > - Do we want to support both TaskGroup & SubDags?
> > > - Process:
> > > - When should we defer the in-scope items to post-2.0
> > > - Completion by a date?
> > > - Progress by a date?
> > > - Progress, Current Work & Discussions
> > > - Scheduler HA
> > > - Docs Improvements
> > > - Helm Chart
> > > - Discuss the issue with sources
> > >
> > >
> > >
> > > Regards,
> > > Kaxil
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
Re: [Meeting Notes] Airflow 2.0 Dev Call #3 - 7 Sep 2020
Posted by Vikram Koka <vi...@astronomer.io>.
Thanks Kaxil, this looks right to me as well.
I updated the main Airflow 2.0 planning page as well to reflect the current
scope and milestones based on this meeting.
*Doc Link*:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning
I also wanted to thank Kevin from the AirBnb team for joining the call on
Monday and offering to help with SmartSensors support as needed. I am
excited that this feature will be part of the 2.0 release.
Vikram
On Wed, Sep 9, 2020 at 3:41 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> LGTM! Thanks, Kaxil for putting this together. It is really helpful!
>
> On Wed, Sep 9, 2020 at 12:29 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Hi all,
> >
> > I have created a document to summarize the discussion from our third dev
> > call for Airflow 2.0.
> >
> > Thank you all who joined the call.
> >
> > *Doc Link*:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#3:7Sep2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020>
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020
> >
> >
> > To all those who attended, can you please double-check and add if I have
> > missed anything?
> >
> > To all those who didn't join, if you disagree to anything in the Summary
> > please
> > voice your opinion.
> >
> > Including the Summary here too (might potentially break formatting):
> >
> > *Key Decisions*
> >
> > - *Smart Sensors*
> > - Will be included in 2.0 as an *early-access* feature with a clear
> > note that *this feature might potentially change in future Airflow
> > version with breaking changes*.
> > - Airbnb team would be happy to help on the support side answering
> > questions related to Smart Sensor.
> > - Add docs around different execution modes for Sensor: *Poke
> > mode*, *Reschedule
> > mode* vs *Smart Sensor*
> > - *Providers Packages*
> > - We had a consensus on *releasing providers packages separately*
> > mainly
> > because of the following reasons:
> > - Separate cadence for providers compared to Airflow, so bugs in
> > operator/hooks can be fixed lot faster.
> > - Enterprises generally would not like to upgrade the “core”
> > (Scheduler) as a small bug can break the deployment and
> > affect all the DAGs
> > - Breaking library changes (new version of a library) can be
> fixed
> > with a new version of Backport/Providers
> > - Upgrades of backport providers are not “that” destructive i.e.
> > even if you upgrade to a newer version and find a bug, you
> > could go back to
> > the previous version without causing any issues at all.
> > - Open questions / Action Items:
> > - How would users figure out “breaking changes” with CALVER
> > Versioning (which is very clear with SEMVER)?
> > - Use plugin Mechanism to:
> > - Register Connections from an external provider to allow
> > custom field or hide existing form fields.
> > - Register Operator Extra links
> > <
> > https://airflow.apache.org/docs/stable/howto/define_extra_link.html>
> > for
> > operators in providers so that a change is not required in
> > Airflow
> > - Backport providers will only be supported/released for *three
> > months after 2.0.0 released*
> > - *Timeline to Airflow 2.0*
> > - Only *critical fixes* (fixes to bugs that takedown Production
> > system) will be backported to 1.10.x core for *six months* after
> > Airflow 2.0 is released.
> >
> > Date
> >
> > Milestone
> >
> > Week of 7 Sep 2020 Create the 2.0.0-test branch
> >
> > While the scope is fluid, we would be rebasing this test branch from
> > master. After we completely freeze the scope, we would only cherrypick
> > commits from Airflow Master to v2-0-test branch if they are “in-scope”.
> > Normal development would continue on Master branch i.e. PRs would be
> > created against Airflow Master.
> > Week of 28 Sep 2020 Cut Functionally complete 2.0 alpha release
> > Week of 12 Oct 2020 Cut first 2.0 beta release
> >
> > Beta snapshots would be published to the Airflow Community to test and
> > create issues to make sure Airflow is functioning and backwards
> compatible.
> > Week of 19 Oct 2020 Cut bridge release based on 1.10.x - jump-off point
> to
> > 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0
> > Week of 26 Oct 2020 Cut second 2.0 beta release
> > Week of 9 Nov 2020 Cut third 2.0 beta release
> > Week of 23 Nov 2020 Cut first 2.0 release candidate (2.0.0rc1)
> >
> > *Things to Discuss Next*
> >
> > - *14 September (Subject to change)*
> > - API
> > - Progress, Current Work & Discussions
> > - Any open questions?
> > - Improvements to SubDags / Concept of TaskGroup
> > - *AIP-34*:
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator
> >
> > - *PR*: https://github.com/apache/airflow/pull/10153
> > - Any concerns on including this PR to 2.0 ??
> > - Do we want to support both TaskGroup & SubDags?
> > - Process:
> > - When should we defer the in-scope items to post-2.0
> > - Completion by a date?
> > - Progress by a date?
> > - Progress, Current Work & Discussions
> > - Scheduler HA
> > - Docs Improvements
> > - Helm Chart
> > - Discuss the issue with sources
> >
> >
> >
> > Regards,
> > Kaxil
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [Meeting Notes] Airflow 2.0 Dev Call #3 - 7 Sep 2020
Posted by Jarek Potiuk <Ja...@polidea.com>.
LGTM! Thanks, Kaxil for putting this together. It is really helpful!
On Wed, Sep 9, 2020 at 12:29 PM Kaxil Naik <ka...@gmail.com> wrote:
> Hi all,
>
> I have created a document to summarize the discussion from our third dev
> call for Airflow 2.0.
>
> Thank you all who joined the call.
>
> *Doc Link*:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#3:7Sep2020
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020>
>
> To all those who attended, can you please double-check and add if I have
> missed anything?
>
> To all those who didn't join, if you disagree to anything in the Summary
> please
> voice your opinion.
>
> Including the Summary here too (might potentially break formatting):
>
> *Key Decisions*
>
> - *Smart Sensors*
> - Will be included in 2.0 as an *early-access* feature with a clear
> note that *this feature might potentially change in future Airflow
> version with breaking changes*.
> - Airbnb team would be happy to help on the support side answering
> questions related to Smart Sensor.
> - Add docs around different execution modes for Sensor: *Poke
> mode*, *Reschedule
> mode* vs *Smart Sensor*
> - *Providers Packages*
> - We had a consensus on *releasing providers packages separately*
> mainly
> because of the following reasons:
> - Separate cadence for providers compared to Airflow, so bugs in
> operator/hooks can be fixed lot faster.
> - Enterprises generally would not like to upgrade the “core”
> (Scheduler) as a small bug can break the deployment and
> affect all the DAGs
> - Breaking library changes (new version of a library) can be fixed
> with a new version of Backport/Providers
> - Upgrades of backport providers are not “that” destructive i.e.
> even if you upgrade to a newer version and find a bug, you
> could go back to
> the previous version without causing any issues at all.
> - Open questions / Action Items:
> - How would users figure out “breaking changes” with CALVER
> Versioning (which is very clear with SEMVER)?
> - Use plugin Mechanism to:
> - Register Connections from an external provider to allow
> custom field or hide existing form fields.
> - Register Operator Extra links
> <
> https://airflow.apache.org/docs/stable/howto/define_extra_link.html>
> for
> operators in providers so that a change is not required in
> Airflow
> - Backport providers will only be supported/released for *three
> months after 2.0.0 released*
> - *Timeline to Airflow 2.0*
> - Only *critical fixes* (fixes to bugs that takedown Production
> system) will be backported to 1.10.x core for *six months* after
> Airflow 2.0 is released.
>
> Date
>
> Milestone
>
> Week of 7 Sep 2020 Create the 2.0.0-test branch
>
> While the scope is fluid, we would be rebasing this test branch from
> master. After we completely freeze the scope, we would only cherrypick
> commits from Airflow Master to v2-0-test branch if they are “in-scope”.
> Normal development would continue on Master branch i.e. PRs would be
> created against Airflow Master.
> Week of 28 Sep 2020 Cut Functionally complete 2.0 alpha release
> Week of 12 Oct 2020 Cut first 2.0 beta release
>
> Beta snapshots would be published to the Airflow Community to test and
> create issues to make sure Airflow is functioning and backwards compatible.
> Week of 19 Oct 2020 Cut bridge release based on 1.10.x - jump-off point to
> 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0
> Week of 26 Oct 2020 Cut second 2.0 beta release
> Week of 9 Nov 2020 Cut third 2.0 beta release
> Week of 23 Nov 2020 Cut first 2.0 release candidate (2.0.0rc1)
>
> *Things to Discuss Next*
>
> - *14 September (Subject to change)*
> - API
> - Progress, Current Work & Discussions
> - Any open questions?
> - Improvements to SubDags / Concept of TaskGroup
> - *AIP-34*:
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator
>
> - *PR*: https://github.com/apache/airflow/pull/10153
> - Any concerns on including this PR to 2.0 ??
> - Do we want to support both TaskGroup & SubDags?
> - Process:
> - When should we defer the in-scope items to post-2.0
> - Completion by a date?
> - Progress by a date?
> - Progress, Current Work & Discussions
> - Scheduler HA
> - Docs Improvements
> - Helm Chart
> - Discuss the issue with sources
>
>
>
> Regards,
> Kaxil
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>