You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2020/09/11 11:56:42 UTC
[Airflow 2.0] State of the HA Scheduler - AIP-15 and
Astronomer/Community plans?
I started to feel that we need to clarify statements about the HA Scheduler
for our community. Not that I am losing sleep regularly over this but it
did keep me away last night when I started to think about it :).
I have a feeling that while we already defined some - rather aggressive -
timelines for 2.0, the subject of HA Scheduler was not touched in the
previous Airflow 2.0 meetings. We are not very far from the release but the
HA scheduler is implemented inside Astronomer and we have not seen any code
for it yet in the community. I understand that a lot of work (not only
development but especially testing) has been put into it from the
Astronomer team internally.
I am actually quite OK with that to be like that. I think Astronomer is a
super-valuable member of the community and I have no doubts Ash and Kaxil
and Daniel and others will do an awesome job with it. I am simply afraid
that when we see it, some of the cases that we see as needed by the
community will require more work. This will either delay the 2.0 release or
we will have to drop it from the 2.0 release. Looking at the number of
discussions we had with - much simpler IMHO - Smart Sensors, I have the
feeling that HA scheduler will spark even more discussions. The AIP-15
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler>
was
not very rich in detail and the last time it has been updated was March
2019 (!) and I have no doubt a big number of design decisions,
observations, learning has happened in Astronomer since.
And to be perfectly honest - I am ok with both of the scenarios I can see:
1) We release HA Scheduler in 2.0
For that, I think we should start looking at the code and discuss it
already quite some time ago IMHO. It might be too late if we want to fit
the aggressive timeline we have - especially that there are other things
the most active people are committing to for 2.0 and they might simply not
have enough time to make the quality review rounds and discussions. I think
we need to see it first to be even able to assess if we think we can make
it within the timeline.
2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer will
use the HA Scheduler in their own service as a "commercial" add-on or
"advantage" of their offering.
In the meantime - between 2.0 and 2.1 Astronomer could donate the code and
we could make sure it is reviewed and merged in the way that answers the
needs of different community members. This has also numerous advantages to
the community - similar to the case of Smart Sensors, Astronomer can test
it in production then and solve all the teething problems of such a
service.
I cannot speak for the business models of Astronomer of course :), but it
seems to me like a nice advantage to have for a while, from the business
point of view. And as a community, we also benefit that we have such a
strong member of the community with a sustainable and good business model.
Without Astronomer's generous support, Ash, Kaxil, and Daniel especially
(but also others) - Airflow would not be where it is today. And I would be
100% happy with such an approach as a PMC and member of the community and I
support it a lot if Astronomer chooses this path.
I think, however, it's the highest time that we decide and clearly
communicate it to the users as a community. At least I have a feeling that
without the community members, committers, and some heavy users being
involved in the open, and having time for quality review and discussion,
releasing HA in 2.0 might be not possible. And to just reiterate - this has
nothing to do with the expected quality of the code and testing, but more
about potential differences in expectations, assumptions, understanding,
performance limitations, and anything else that might (and usually does)
come up.
I think - since we already started to publish the schedule, this is the
right time that we make a decision on that and align expectations.
Ry, Vikram - I'd love to hear what the intentions of Astronomer as a
company for the HA Scheduler are? I know as a group of committers we said
it a number of times that HA Scheduler will be in 2.0 so we built the
expectations among our users as a community. But maybe you really think
that pursuing scenario 2) (or maybe another scenario I have not thought
about) is the way to go for Astronomer?
As I wrote above - I am personally perfectly fine with either of the
scenarios, and I think they are both beneficial for the community, but I
think we should discuss it, align expectations, and clearly communicate as
the Apache Airflow community.
J.
--
Jarek Potiuk
Polidea | Principal Software Engineer
M: +48 660 796 129
Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and
Astronomer/Community plans?
Posted by Jarek Potiuk <Ja...@polidea.com>.
Actually :) the "Valid" link was marked as "Original AIP" in the doc I
linked :). I will fix it.
On Fri, Sep 11, 2020 at 2:14 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Ah. Thanks! It just popped as the "first" when I looked at it I will mark
> it as [ARCHIVED} and move to the archive.
> So my point about updates are not really valid :). It's rather detailed
> and updated from what I see.
>
> J.
>
> On Fri, Sep 11, 2020 at 2:08 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Just a note here:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651
>> <
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651>
>> is
>> the correct link for the AIP proposed for Scheduler HA. The other link was
>> an old proposal from someone else.
>>
>> On Fri, Sep 11, 2020 at 12:57 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> > I started to feel that we need to clarify statements about the HA
>> Scheduler
>> > for our community. Not that I am losing sleep regularly over this but it
>> > did keep me away last night when I started to think about it :).
>> >
>> > I have a feeling that while we already defined some - rather aggressive
>> -
>> > timelines for 2.0, the subject of HA Scheduler was not touched in the
>> > previous Airflow 2.0 meetings. We are not very far from the release but
>> the
>> > HA scheduler is implemented inside Astronomer and we have not seen any
>> code
>> > for it yet in the community. I understand that a lot of work (not only
>> > development but especially testing) has been put into it from the
>> > Astronomer team internally.
>> >
>> > I am actually quite OK with that to be like that. I think Astronomer is
>> a
>> > super-valuable member of the community and I have no doubts Ash and
>> Kaxil
>> > and Daniel and others will do an awesome job with it. I am simply afraid
>> > that when we see it, some of the cases that we see as needed by the
>> > community will require more work. This will either delay the 2.0
>> release or
>> > we will have to drop it from the 2.0 release. Looking at the number of
>> > discussions we had with - much simpler IMHO - Smart Sensors, I have the
>> > feeling that HA scheduler will spark even more discussions. The AIP-15
>> > <
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler
>> > >
>> > was
>> > not very rich in detail and the last time it has been updated was March
>> > 2019 (!) and I have no doubt a big number of design decisions,
>> > observations, learning has happened in Astronomer since.
>> >
>> > And to be perfectly honest - I am ok with both of the scenarios I can
>> see:
>> >
>> > 1) We release HA Scheduler in 2.0
>> >
>> > For that, I think we should start looking at the code and discuss it
>> > already quite some time ago IMHO. It might be too late if we want to fit
>> > the aggressive timeline we have - especially that there are other things
>> > the most active people are committing to for 2.0 and they might simply
>> not
>> > have enough time to make the quality review rounds and discussions. I
>> think
>> > we need to see it first to be even able to assess if we think we can
>> make
>> > it within the timeline.
>> >
>> > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer
>> will
>> > use the HA Scheduler in their own service as a "commercial" add-on or
>> > "advantage" of their offering.
>> >
>> > In the meantime - between 2.0 and 2.1 Astronomer could donate the code
>> and
>> > we could make sure it is reviewed and merged in the way that answers the
>> > needs of different community members. This has also numerous advantages
>> to
>> > the community - similar to the case of Smart Sensors, Astronomer can
>> test
>> > it in production then and solve all the teething problems of such a
>> > service.
>> >
>> > I cannot speak for the business models of Astronomer of course :), but
>> it
>> > seems to me like a nice advantage to have for a while, from the business
>> > point of view. And as a community, we also benefit that we have such a
>> > strong member of the community with a sustainable and good business
>> model.
>> > Without Astronomer's generous support, Ash, Kaxil, and Daniel especially
>> > (but also others) - Airflow would not be where it is today. And I would
>> be
>> > 100% happy with such an approach as a PMC and member of the community
>> and I
>> > support it a lot if Astronomer chooses this path.
>> >
>> > I think, however, it's the highest time that we decide and clearly
>> > communicate it to the users as a community. At least I have a feeling
>> that
>> > without the community members, committers, and some heavy users being
>> > involved in the open, and having time for quality review and discussion,
>> > releasing HA in 2.0 might be not possible. And to just reiterate - this
>> has
>> > nothing to do with the expected quality of the code and testing, but
>> more
>> > about potential differences in expectations, assumptions, understanding,
>> > performance limitations, and anything else that might (and usually does)
>> > come up.
>> >
>> > I think - since we already started to publish the schedule, this is the
>> > right time that we make a decision on that and align expectations.
>> >
>> > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a
>> > company for the HA Scheduler are? I know as a group of committers we
>> said
>> > it a number of times that HA Scheduler will be in 2.0 so we built the
>> > expectations among our users as a community. But maybe you really think
>> > that pursuing scenario 2) (or maybe another scenario I have not thought
>> > about) is the way to go for Astronomer?
>> >
>> > As I wrote above - I am personally perfectly fine with either of the
>> > scenarios, and I think they are both beneficial for the community, but I
>> > think we should discuss it, align expectations, and clearly communicate
>> as
>> > the Apache Airflow community.
>> >
>> > J.
>> >
>> > --
>> > Jarek Potiuk
>> > Polidea | Principal Software Engineer
>> >
>> > M: +48 660 796 129
>> >
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and
Astronomer/Community plans?
Posted by Jarek Potiuk <Ja...@polidea.com>.
Ah. Thanks! It just popped as the "first" when I looked at it I will mark
it as [ARCHIVED} and move to the archive.
So my point about updates are not really valid :). It's rather detailed and
updated from what I see.
J.
On Fri, Sep 11, 2020 at 2:08 PM Kaxil Naik <ka...@gmail.com> wrote:
> Just a note here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651
> <
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651>
> is
> the correct link for the AIP proposed for Scheduler HA. The other link was
> an old proposal from someone else.
>
> On Fri, Sep 11, 2020 at 12:57 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > I started to feel that we need to clarify statements about the HA
> Scheduler
> > for our community. Not that I am losing sleep regularly over this but it
> > did keep me away last night when I started to think about it :).
> >
> > I have a feeling that while we already defined some - rather aggressive -
> > timelines for 2.0, the subject of HA Scheduler was not touched in the
> > previous Airflow 2.0 meetings. We are not very far from the release but
> the
> > HA scheduler is implemented inside Astronomer and we have not seen any
> code
> > for it yet in the community. I understand that a lot of work (not only
> > development but especially testing) has been put into it from the
> > Astronomer team internally.
> >
> > I am actually quite OK with that to be like that. I think Astronomer is a
> > super-valuable member of the community and I have no doubts Ash and Kaxil
> > and Daniel and others will do an awesome job with it. I am simply afraid
> > that when we see it, some of the cases that we see as needed by the
> > community will require more work. This will either delay the 2.0 release
> or
> > we will have to drop it from the 2.0 release. Looking at the number of
> > discussions we had with - much simpler IMHO - Smart Sensors, I have the
> > feeling that HA scheduler will spark even more discussions. The AIP-15
> > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler
> > >
> > was
> > not very rich in detail and the last time it has been updated was March
> > 2019 (!) and I have no doubt a big number of design decisions,
> > observations, learning has happened in Astronomer since.
> >
> > And to be perfectly honest - I am ok with both of the scenarios I can
> see:
> >
> > 1) We release HA Scheduler in 2.0
> >
> > For that, I think we should start looking at the code and discuss it
> > already quite some time ago IMHO. It might be too late if we want to fit
> > the aggressive timeline we have - especially that there are other things
> > the most active people are committing to for 2.0 and they might simply
> not
> > have enough time to make the quality review rounds and discussions. I
> think
> > we need to see it first to be even able to assess if we think we can make
> > it within the timeline.
> >
> > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer
> will
> > use the HA Scheduler in their own service as a "commercial" add-on or
> > "advantage" of their offering.
> >
> > In the meantime - between 2.0 and 2.1 Astronomer could donate the code
> and
> > we could make sure it is reviewed and merged in the way that answers the
> > needs of different community members. This has also numerous advantages
> to
> > the community - similar to the case of Smart Sensors, Astronomer can test
> > it in production then and solve all the teething problems of such a
> > service.
> >
> > I cannot speak for the business models of Astronomer of course :), but it
> > seems to me like a nice advantage to have for a while, from the business
> > point of view. And as a community, we also benefit that we have such a
> > strong member of the community with a sustainable and good business
> model.
> > Without Astronomer's generous support, Ash, Kaxil, and Daniel especially
> > (but also others) - Airflow would not be where it is today. And I would
> be
> > 100% happy with such an approach as a PMC and member of the community
> and I
> > support it a lot if Astronomer chooses this path.
> >
> > I think, however, it's the highest time that we decide and clearly
> > communicate it to the users as a community. At least I have a feeling
> that
> > without the community members, committers, and some heavy users being
> > involved in the open, and having time for quality review and discussion,
> > releasing HA in 2.0 might be not possible. And to just reiterate - this
> has
> > nothing to do with the expected quality of the code and testing, but more
> > about potential differences in expectations, assumptions, understanding,
> > performance limitations, and anything else that might (and usually does)
> > come up.
> >
> > I think - since we already started to publish the schedule, this is the
> > right time that we make a decision on that and align expectations.
> >
> > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a
> > company for the HA Scheduler are? I know as a group of committers we said
> > it a number of times that HA Scheduler will be in 2.0 so we built the
> > expectations among our users as a community. But maybe you really think
> > that pursuing scenario 2) (or maybe another scenario I have not thought
> > about) is the way to go for Astronomer?
> >
> > As I wrote above - I am personally perfectly fine with either of the
> > scenarios, and I think they are both beneficial for the community, but I
> > think we should discuss it, align expectations, and clearly communicate
> as
> > the Apache Airflow community.
> >
> > J.
> >
> > --
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and
Astronomer/Community plans?
Posted by Kaxil Naik <ka...@gmail.com>.
Just a note here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651> is
the correct link for the AIP proposed for Scheduler HA. The other link was
an old proposal from someone else.
On Fri, Sep 11, 2020 at 12:57 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I started to feel that we need to clarify statements about the HA Scheduler
> for our community. Not that I am losing sleep regularly over this but it
> did keep me away last night when I started to think about it :).
>
> I have a feeling that while we already defined some - rather aggressive -
> timelines for 2.0, the subject of HA Scheduler was not touched in the
> previous Airflow 2.0 meetings. We are not very far from the release but the
> HA scheduler is implemented inside Astronomer and we have not seen any code
> for it yet in the community. I understand that a lot of work (not only
> development but especially testing) has been put into it from the
> Astronomer team internally.
>
> I am actually quite OK with that to be like that. I think Astronomer is a
> super-valuable member of the community and I have no doubts Ash and Kaxil
> and Daniel and others will do an awesome job with it. I am simply afraid
> that when we see it, some of the cases that we see as needed by the
> community will require more work. This will either delay the 2.0 release or
> we will have to drop it from the 2.0 release. Looking at the number of
> discussions we had with - much simpler IMHO - Smart Sensors, I have the
> feeling that HA scheduler will spark even more discussions. The AIP-15
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler
> >
> was
> not very rich in detail and the last time it has been updated was March
> 2019 (!) and I have no doubt a big number of design decisions,
> observations, learning has happened in Astronomer since.
>
> And to be perfectly honest - I am ok with both of the scenarios I can see:
>
> 1) We release HA Scheduler in 2.0
>
> For that, I think we should start looking at the code and discuss it
> already quite some time ago IMHO. It might be too late if we want to fit
> the aggressive timeline we have - especially that there are other things
> the most active people are committing to for 2.0 and they might simply not
> have enough time to make the quality review rounds and discussions. I think
> we need to see it first to be even able to assess if we think we can make
> it within the timeline.
>
> 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer will
> use the HA Scheduler in their own service as a "commercial" add-on or
> "advantage" of their offering.
>
> In the meantime - between 2.0 and 2.1 Astronomer could donate the code and
> we could make sure it is reviewed and merged in the way that answers the
> needs of different community members. This has also numerous advantages to
> the community - similar to the case of Smart Sensors, Astronomer can test
> it in production then and solve all the teething problems of such a
> service.
>
> I cannot speak for the business models of Astronomer of course :), but it
> seems to me like a nice advantage to have for a while, from the business
> point of view. And as a community, we also benefit that we have such a
> strong member of the community with a sustainable and good business model.
> Without Astronomer's generous support, Ash, Kaxil, and Daniel especially
> (but also others) - Airflow would not be where it is today. And I would be
> 100% happy with such an approach as a PMC and member of the community and I
> support it a lot if Astronomer chooses this path.
>
> I think, however, it's the highest time that we decide and clearly
> communicate it to the users as a community. At least I have a feeling that
> without the community members, committers, and some heavy users being
> involved in the open, and having time for quality review and discussion,
> releasing HA in 2.0 might be not possible. And to just reiterate - this has
> nothing to do with the expected quality of the code and testing, but more
> about potential differences in expectations, assumptions, understanding,
> performance limitations, and anything else that might (and usually does)
> come up.
>
> I think - since we already started to publish the schedule, this is the
> right time that we make a decision on that and align expectations.
>
> Ry, Vikram - I'd love to hear what the intentions of Astronomer as a
> company for the HA Scheduler are? I know as a group of committers we said
> it a number of times that HA Scheduler will be in 2.0 so we built the
> expectations among our users as a community. But maybe you really think
> that pursuing scenario 2) (or maybe another scenario I have not thought
> about) is the way to go for Astronomer?
>
> As I wrote above - I am personally perfectly fine with either of the
> scenarios, and I think they are both beneficial for the community, but I
> think we should discuss it, align expectations, and clearly communicate as
> the Apache Airflow community.
>
> J.
>
> --
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>
Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and
Astronomer/Community plans?
Posted by Jarek Potiuk <Ja...@polidea.com>.
This is really great. It was never my intention to question
the viability of the HA scheduler for the 2.0 - if that was misunderstood -
true apologies.
I certainly appreciate the hard work done on it by the Astronomer team - I
just wanted to stress that - unlike most other features of Airflow, this
one is not yet out in the community to comment, discuss and iterate on and
I was really afraid that if someone raises big concerns about the
implementation (not because of quality of lack of tests or benchmarks, but
because others can have valid concerns about the architecture, side effect,
assumptions, it might simply be very little time to address those in the
usual way we approach them in the community.
It often happens that when more eyes look at the changes, we simply
strat realising that we should address something else. I think if we want
to do everything humanly possible to make it into 2.0 - we should simply
start looking at it together with the community.
Once again I think I could be misunderstood, I was not complaining nor
calling for delaying nor doubting the possibility of implementing it by the
Astronomer team (with a lot of effort already put there). Simply - from
what I see so far - most of the big features like that when first proposed,
often cause a lot of discussion, hundreds of comments, doubts and possibly
resulting changes when the broader community looks at it.
J.
On Fri, Sep 11, 2020 at 7:06 PM Kaxil Naik <ka...@gmail.com> wrote:
> Hi Jarek and all,
>
> Scheduler HA was already on Agenda
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-ThingstoDiscussNext
> >
> for next Monday's Airflow 2.0 DEV call where we could discuss it in detail
> but I definitely think we should get Scheduler HA in 2.0.
>
> In our very first 2.0 Dev call we all agreed that we would be willing to
> delay the release if needed by a few weeks for the following features:
>
> - Airflow REST API
> - Scheduler HA
> - Docker Image & Helm Chart
> - Providers Packages
>
> One of the crucial benefits of the PRs that Ash and myself will create in
> upcoming days as part of Scheduler HA work would also be around
> optimizations, the preliminary numbers for which Ash had shared in the
> Airflow Summit.
>
> Based on the current timeline we agreed on the last call, we almost have
> around 2 months of Testing (including Alpha & Beta -- Oct - Nov) period
> that
> should aid in figuring or fixing out any confusion, bugs or issues we
> encounter. Without Scheduler HA we won't need that large of a testing cycle
> of two months and if
> after that two months of testing it doesn't work we'll back it out and
> remove it from the release branch.
>
> Ash & I would be working on it full-time and would be our number one
> priority to get that in.
>
> It will also help in Airflow 2.0 story of Scalability and Reliability.
>
> Regards,
> Kaxil
>
>
>
> On Fri, Sep 11, 2020 at 3:15 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > I am personally even super happy if Astronomer provides it to the
> customers
> > with commercial obligations - until it is merged in 2.1 for example.
> > Including the support - while we are discussing it and merging and
> > releasing it in 2.1 (and likely later supporting migration to the
> community
> > one internally).
> >
> > I believe there is nothing to prevent that from the ASF rules (and
> > community) point of view :). It just has to be transparently
> communicated,
> > that's all :).
> >
> > J.
> >
> >
> > On Fri, Sep 11, 2020 at 2:18 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > Hi Jarek, and all.
> > >
> > > You aren't the only one to have this thought -- it's been on my mind
> too.
> > >
> > > Sadly I wasn't able to get the code in a PR-able state before heading
> > > off on paternity leave. I have started separating out and submitting
> the
> > > "easy"/preparatory PRs to try to lessen the size of the "main" PR:
> > >
> > > https://github.com/apache/airflow/pull/10729
> > > https://github.com/apache/airflow/pull/10710
> > > https://github.com/apache/airflow/pull/10706
> > >
> > > But yes, at some point it needs a large "big-bang" PR. That is coming.
> > >
> > > We will have a discussion internally and see what course we think is
> > > best for us to take. One possible path is we submit the PR to open
> > > discussion, and concurrently make that change available via our
> > > astronomer images of 2.0 (which is available under Apache 2 License
> > > without commercial obligations, so usable by anyone in the community)
> > >
> > > Thanks for bringing this up.
> > >
> > > -ash
> > >
> > > On Sep 11 2020, at 12:56 pm, Jarek Potiuk <Ja...@polidea.com>
> > > wrote:
> > >
> > > > I started to feel that we need to clarify statements about the HA
> > > Scheduler
> > > > for our community. Not that I am losing sleep regularly over this but
> > it
> > > > did keep me away last night when I started to think about it :).
> > > >
> > > > I have a feeling that while we already defined some - rather
> > > > aggressive -
> > > > timelines for 2.0, the subject of HA Scheduler was not touched in the
> > > > previous Airflow 2.0 meetings. We are not very far from the release
> > > > but the
> > > > HA scheduler is implemented inside Astronomer and we have not seen
> any
> > > code
> > > > for it yet in the community. I understand that a lot of work (not
> only
> > > > development but especially testing) has been put into it from the
> > > > Astronomer team internally.
> > > >
> > > > I am actually quite OK with that to be like that. I think Astronomer
> > > > is a
> > > > super-valuable member of the community and I have no doubts Ash and
> > Kaxil
> > > > and Daniel and others will do an awesome job with it. I am simply
> > afraid
> > > > that when we see it, some of the cases that we see as needed by the
> > > > community will require more work. This will either delay the 2.0
> > > > release or
> > > > we will have to drop it from the 2.0 release. Looking at the number
> of
> > > > discussions we had with - much simpler IMHO - Smart Sensors, I have
> the
> > > > feeling that HA scheduler will spark even more discussions. The
> AIP-15
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler
> > > >
> > > > was
> > > > not very rich in detail and the last time it has been updated was
> March
> > > > 2019 (!) and I have no doubt a big number of design decisions,
> > > > observations, learning has happened in Astronomer since.
> > > >
> > > > And to be perfectly honest - I am ok with both of the scenarios I can
> > > see:
> > > >
> > > > 1) We release HA Scheduler in 2.0
> > > >
> > > > For that, I think we should start looking at the code and discuss it
> > > > already quite some time ago IMHO. It might be too late if we want to
> > fit
> > > > the aggressive timeline we have - especially that there are other
> > things
> > > > the most active people are committing to for 2.0 and they might
> simply
> > > not
> > > > have enough time to make the quality review rounds and discussions. I
> > > think
> > > > we need to see it first to be even able to assess if we think we can
> > make
> > > > it within the timeline.
> > > >
> > > > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and
> Astronomer
> > > will
> > > > use the HA Scheduler in their own service as a "commercial" add-on or
> > > > "advantage" of their offering.
> > > >
> > > > In the meantime - between 2.0 and 2.1 Astronomer could donate the
> code
> > > and
> > > > we could make sure it is reviewed and merged in the way that answers
> > the
> > > > needs of different community members. This has also numerous
> > > > advantages to
> > > > the community - similar to the case of Smart Sensors, Astronomer can
> > test
> > > > it in production then and solve all the teething problems of such a
> > > > service.
> > > >
> > > > I cannot speak for the business models of Astronomer of course :),
> but
> > it
> > > > seems to me like a nice advantage to have for a while, from the
> > business
> > > > point of view. And as a community, we also benefit that we have such
> a
> > > > strong member of the community with a sustainable and good business
> > > model.
> > > > Without Astronomer's generous support, Ash, Kaxil, and Daniel
> > especially
> > > > (but also others) - Airflow would not be where it is today. And I
> > > > would be
> > > > 100% happy with such an approach as a PMC and member of the community
> > > > and I
> > > > support it a lot if Astronomer chooses this path.
> > > >
> > > > I think, however, it's the highest time that we decide and clearly
> > > > communicate it to the users as a community. At least I have a feeling
> > > that
> > > > without the community members, committers, and some heavy users being
> > > > involved in the open, and having time for quality review and
> > discussion,
> > > > releasing HA in 2.0 might be not possible. And to just reiterate -
> > > > this has
> > > > nothing to do with the expected quality of the code and testing, but
> > more
> > > > about potential differences in expectations, assumptions,
> > understanding,
> > > > performance limitations, and anything else that might (and usually
> > does)
> > > > come up.
> > > >
> > > > I think - since we already started to publish the schedule, this is
> the
> > > > right time that we make a decision on that and align expectations.
> > > >
> > > > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a
> > > > company for the HA Scheduler are? I know as a group of committers we
> > said
> > > > it a number of times that HA Scheduler will be in 2.0 so we built the
> > > > expectations among our users as a community. But maybe you really
> think
> > > > that pursuing scenario 2) (or maybe another scenario I have not
> thought
> > > > about) is the way to go for Astronomer?
> > > >
> > > > As I wrote above - I am personally perfectly fine with either of the
> > > > scenarios, and I think they are both beneficial for the community,
> but
> > I
> > > > think we should discuss it, align expectations, and clearly
> > > > communicate as
> > > > the Apache Airflow community.
> > > >
> > > > J.
> > > >
> > > > --
> > > > Jarek Potiuk
> > > > Polidea | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and
Astronomer/Community plans?
Posted by Kaxil Naik <ka...@gmail.com>.
Hi Jarek and all,
Scheduler HA was already on Agenda
<https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-ThingstoDiscussNext>
for next Monday's Airflow 2.0 DEV call where we could discuss it in detail
but I definitely think we should get Scheduler HA in 2.0.
In our very first 2.0 Dev call we all agreed that we would be willing to
delay the release if needed by a few weeks for the following features:
- Airflow REST API
- Scheduler HA
- Docker Image & Helm Chart
- Providers Packages
One of the crucial benefits of the PRs that Ash and myself will create in
upcoming days as part of Scheduler HA work would also be around
optimizations, the preliminary numbers for which Ash had shared in the
Airflow Summit.
Based on the current timeline we agreed on the last call, we almost have
around 2 months of Testing (including Alpha & Beta -- Oct - Nov) period that
should aid in figuring or fixing out any confusion, bugs or issues we
encounter. Without Scheduler HA we won't need that large of a testing cycle
of two months and if
after that two months of testing it doesn't work we'll back it out and
remove it from the release branch.
Ash & I would be working on it full-time and would be our number one
priority to get that in.
It will also help in Airflow 2.0 story of Scalability and Reliability.
Regards,
Kaxil
On Fri, Sep 11, 2020 at 3:15 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I am personally even super happy if Astronomer provides it to the customers
> with commercial obligations - until it is merged in 2.1 for example.
> Including the support - while we are discussing it and merging and
> releasing it in 2.1 (and likely later supporting migration to the community
> one internally).
>
> I believe there is nothing to prevent that from the ASF rules (and
> community) point of view :). It just has to be transparently communicated,
> that's all :).
>
> J.
>
>
> On Fri, Sep 11, 2020 at 2:18 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > Hi Jarek, and all.
> >
> > You aren't the only one to have this thought -- it's been on my mind too.
> >
> > Sadly I wasn't able to get the code in a PR-able state before heading
> > off on paternity leave. I have started separating out and submitting the
> > "easy"/preparatory PRs to try to lessen the size of the "main" PR:
> >
> > https://github.com/apache/airflow/pull/10729
> > https://github.com/apache/airflow/pull/10710
> > https://github.com/apache/airflow/pull/10706
> >
> > But yes, at some point it needs a large "big-bang" PR. That is coming.
> >
> > We will have a discussion internally and see what course we think is
> > best for us to take. One possible path is we submit the PR to open
> > discussion, and concurrently make that change available via our
> > astronomer images of 2.0 (which is available under Apache 2 License
> > without commercial obligations, so usable by anyone in the community)
> >
> > Thanks for bringing this up.
> >
> > -ash
> >
> > On Sep 11 2020, at 12:56 pm, Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > I started to feel that we need to clarify statements about the HA
> > Scheduler
> > > for our community. Not that I am losing sleep regularly over this but
> it
> > > did keep me away last night when I started to think about it :).
> > >
> > > I have a feeling that while we already defined some - rather
> > > aggressive -
> > > timelines for 2.0, the subject of HA Scheduler was not touched in the
> > > previous Airflow 2.0 meetings. We are not very far from the release
> > > but the
> > > HA scheduler is implemented inside Astronomer and we have not seen any
> > code
> > > for it yet in the community. I understand that a lot of work (not only
> > > development but especially testing) has been put into it from the
> > > Astronomer team internally.
> > >
> > > I am actually quite OK with that to be like that. I think Astronomer
> > > is a
> > > super-valuable member of the community and I have no doubts Ash and
> Kaxil
> > > and Daniel and others will do an awesome job with it. I am simply
> afraid
> > > that when we see it, some of the cases that we see as needed by the
> > > community will require more work. This will either delay the 2.0
> > > release or
> > > we will have to drop it from the 2.0 release. Looking at the number of
> > > discussions we had with - much simpler IMHO - Smart Sensors, I have the
> > > feeling that HA scheduler will spark even more discussions. The AIP-15
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler
> > >
> > > was
> > > not very rich in detail and the last time it has been updated was March
> > > 2019 (!) and I have no doubt a big number of design decisions,
> > > observations, learning has happened in Astronomer since.
> > >
> > > And to be perfectly honest - I am ok with both of the scenarios I can
> > see:
> > >
> > > 1) We release HA Scheduler in 2.0
> > >
> > > For that, I think we should start looking at the code and discuss it
> > > already quite some time ago IMHO. It might be too late if we want to
> fit
> > > the aggressive timeline we have - especially that there are other
> things
> > > the most active people are committing to for 2.0 and they might simply
> > not
> > > have enough time to make the quality review rounds and discussions. I
> > think
> > > we need to see it first to be even able to assess if we think we can
> make
> > > it within the timeline.
> > >
> > > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer
> > will
> > > use the HA Scheduler in their own service as a "commercial" add-on or
> > > "advantage" of their offering.
> > >
> > > In the meantime - between 2.0 and 2.1 Astronomer could donate the code
> > and
> > > we could make sure it is reviewed and merged in the way that answers
> the
> > > needs of different community members. This has also numerous
> > > advantages to
> > > the community - similar to the case of Smart Sensors, Astronomer can
> test
> > > it in production then and solve all the teething problems of such a
> > > service.
> > >
> > > I cannot speak for the business models of Astronomer of course :), but
> it
> > > seems to me like a nice advantage to have for a while, from the
> business
> > > point of view. And as a community, we also benefit that we have such a
> > > strong member of the community with a sustainable and good business
> > model.
> > > Without Astronomer's generous support, Ash, Kaxil, and Daniel
> especially
> > > (but also others) - Airflow would not be where it is today. And I
> > > would be
> > > 100% happy with such an approach as a PMC and member of the community
> > > and I
> > > support it a lot if Astronomer chooses this path.
> > >
> > > I think, however, it's the highest time that we decide and clearly
> > > communicate it to the users as a community. At least I have a feeling
> > that
> > > without the community members, committers, and some heavy users being
> > > involved in the open, and having time for quality review and
> discussion,
> > > releasing HA in 2.0 might be not possible. And to just reiterate -
> > > this has
> > > nothing to do with the expected quality of the code and testing, but
> more
> > > about potential differences in expectations, assumptions,
> understanding,
> > > performance limitations, and anything else that might (and usually
> does)
> > > come up.
> > >
> > > I think - since we already started to publish the schedule, this is the
> > > right time that we make a decision on that and align expectations.
> > >
> > > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a
> > > company for the HA Scheduler are? I know as a group of committers we
> said
> > > it a number of times that HA Scheduler will be in 2.0 so we built the
> > > expectations among our users as a community. But maybe you really think
> > > that pursuing scenario 2) (or maybe another scenario I have not thought
> > > about) is the way to go for Astronomer?
> > >
> > > As I wrote above - I am personally perfectly fine with either of the
> > > scenarios, and I think they are both beneficial for the community, but
> I
> > > think we should discuss it, align expectations, and clearly
> > > communicate as
> > > the Apache Airflow community.
> > >
> > > J.
> > >
> > > --
> > > Jarek Potiuk
> > > Polidea | Principal Software Engineer
> > >
> > > M: +48 660 796 129
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and
Astronomer/Community plans?
Posted by Vikram Koka <vi...@astronomer.io>.
Jarek,
It has been our intention (as Astronomer) to release the Scheduler HA work
directly to open source as part of Airflow 2.0.
We realized early on that the Scheduler reliability and performance were
highlighted as the key issues from the community as part of the latest
survey results from last year. We therefore crafted the Scheduler HA AIP to
cover those areas explicitly and scalability in addition. We believe that
these are key for Airflow to continue to grow and therefore have been
committed from the get go to make these available as open source as quickly
as possible for the community.
Ash shared his preliminary benchmark numbers as part of the Airflow 2.0
summit, but had to step away after that because of paternity leave. Now
that he is back, we have begun merging back all his changes into master. We
intend to do everything humanly possible to get it into the 2.0 release
within the published timeline.
Best regards,
Vikram
On Fri, Sep 11, 2020 at 7:15 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I am personally even super happy if Astronomer provides it to the customers
> with commercial obligations - until it is merged in 2.1 for example.
> Including the support - while we are discussing it and merging and
> releasing it in 2.1 (and likely later supporting migration to the community
> one internally).
>
> I believe there is nothing to prevent that from the ASF rules (and
> community) point of view :). It just has to be transparently communicated,
> that's all :).
>
> J.
>
>
> On Fri, Sep 11, 2020 at 2:18 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > Hi Jarek, and all.
> >
> > You aren't the only one to have this thought -- it's been on my mind too.
> >
> > Sadly I wasn't able to get the code in a PR-able state before heading
> > off on paternity leave. I have started separating out and submitting the
> > "easy"/preparatory PRs to try to lessen the size of the "main" PR:
> >
> > https://github.com/apache/airflow/pull/10729
> > https://github.com/apache/airflow/pull/10710
> > https://github.com/apache/airflow/pull/10706
> >
> > But yes, at some point it needs a large "big-bang" PR. That is coming.
> >
> > We will have a discussion internally and see what course we think is
> > best for us to take. One possible path is we submit the PR to open
> > discussion, and concurrently make that change available via our
> > astronomer images of 2.0 (which is available under Apache 2 License
> > without commercial obligations, so usable by anyone in the community)
> >
> > Thanks for bringing this up.
> >
> > -ash
> >
> > On Sep 11 2020, at 12:56 pm, Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > I started to feel that we need to clarify statements about the HA
> > Scheduler
> > > for our community. Not that I am losing sleep regularly over this but
> it
> > > did keep me away last night when I started to think about it :).
> > >
> > > I have a feeling that while we already defined some - rather
> > > aggressive -
> > > timelines for 2.0, the subject of HA Scheduler was not touched in the
> > > previous Airflow 2.0 meetings. We are not very far from the release
> > > but the
> > > HA scheduler is implemented inside Astronomer and we have not seen any
> > code
> > > for it yet in the community. I understand that a lot of work (not only
> > > development but especially testing) has been put into it from the
> > > Astronomer team internally.
> > >
> > > I am actually quite OK with that to be like that. I think Astronomer
> > > is a
> > > super-valuable member of the community and I have no doubts Ash and
> Kaxil
> > > and Daniel and others will do an awesome job with it. I am simply
> afraid
> > > that when we see it, some of the cases that we see as needed by the
> > > community will require more work. This will either delay the 2.0
> > > release or
> > > we will have to drop it from the 2.0 release. Looking at the number of
> > > discussions we had with - much simpler IMHO - Smart Sensors, I have the
> > > feeling that HA scheduler will spark even more discussions. The AIP-15
> > > <
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler
> > >
> > > was
> > > not very rich in detail and the last time it has been updated was March
> > > 2019 (!) and I have no doubt a big number of design decisions,
> > > observations, learning has happened in Astronomer since.
> > >
> > > And to be perfectly honest - I am ok with both of the scenarios I can
> > see:
> > >
> > > 1) We release HA Scheduler in 2.0
> > >
> > > For that, I think we should start looking at the code and discuss it
> > > already quite some time ago IMHO. It might be too late if we want to
> fit
> > > the aggressive timeline we have - especially that there are other
> things
> > > the most active people are committing to for 2.0 and they might simply
> > not
> > > have enough time to make the quality review rounds and discussions. I
> > think
> > > we need to see it first to be even able to assess if we think we can
> make
> > > it within the timeline.
> > >
> > > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer
> > will
> > > use the HA Scheduler in their own service as a "commercial" add-on or
> > > "advantage" of their offering.
> > >
> > > In the meantime - between 2.0 and 2.1 Astronomer could donate the code
> > and
> > > we could make sure it is reviewed and merged in the way that answers
> the
> > > needs of different community members. This has also numerous
> > > advantages to
> > > the community - similar to the case of Smart Sensors, Astronomer can
> test
> > > it in production then and solve all the teething problems of such a
> > > service.
> > >
> > > I cannot speak for the business models of Astronomer of course :), but
> it
> > > seems to me like a nice advantage to have for a while, from the
> business
> > > point of view. And as a community, we also benefit that we have such a
> > > strong member of the community with a sustainable and good business
> > model.
> > > Without Astronomer's generous support, Ash, Kaxil, and Daniel
> especially
> > > (but also others) - Airflow would not be where it is today. And I
> > > would be
> > > 100% happy with such an approach as a PMC and member of the community
> > > and I
> > > support it a lot if Astronomer chooses this path.
> > >
> > > I think, however, it's the highest time that we decide and clearly
> > > communicate it to the users as a community. At least I have a feeling
> > that
> > > without the community members, committers, and some heavy users being
> > > involved in the open, and having time for quality review and
> discussion,
> > > releasing HA in 2.0 might be not possible. And to just reiterate -
> > > this has
> > > nothing to do with the expected quality of the code and testing, but
> more
> > > about potential differences in expectations, assumptions,
> understanding,
> > > performance limitations, and anything else that might (and usually
> does)
> > > come up.
> > >
> > > I think - since we already started to publish the schedule, this is the
> > > right time that we make a decision on that and align expectations.
> > >
> > > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a
> > > company for the HA Scheduler are? I know as a group of committers we
> said
> > > it a number of times that HA Scheduler will be in 2.0 so we built the
> > > expectations among our users as a community. But maybe you really think
> > > that pursuing scenario 2) (or maybe another scenario I have not thought
> > > about) is the way to go for Astronomer?
> > >
> > > As I wrote above - I am personally perfectly fine with either of the
> > > scenarios, and I think they are both beneficial for the community, but
> I
> > > think we should discuss it, align expectations, and clearly
> > > communicate as
> > > the Apache Airflow community.
> > >
> > > J.
> > >
> > > --
> > > Jarek Potiuk
> > > Polidea | Principal Software Engineer
> > >
> > > M: +48 660 796 129
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and
Astronomer/Community plans?
Posted by Jarek Potiuk <Ja...@polidea.com>.
I am personally even super happy if Astronomer provides it to the customers
with commercial obligations - until it is merged in 2.1 for example.
Including the support - while we are discussing it and merging and
releasing it in 2.1 (and likely later supporting migration to the community
one internally).
I believe there is nothing to prevent that from the ASF rules (and
community) point of view :). It just has to be transparently communicated,
that's all :).
J.
On Fri, Sep 11, 2020 at 2:18 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> Hi Jarek, and all.
>
> You aren't the only one to have this thought -- it's been on my mind too.
>
> Sadly I wasn't able to get the code in a PR-able state before heading
> off on paternity leave. I have started separating out and submitting the
> "easy"/preparatory PRs to try to lessen the size of the "main" PR:
>
> https://github.com/apache/airflow/pull/10729
> https://github.com/apache/airflow/pull/10710
> https://github.com/apache/airflow/pull/10706
>
> But yes, at some point it needs a large "big-bang" PR. That is coming.
>
> We will have a discussion internally and see what course we think is
> best for us to take. One possible path is we submit the PR to open
> discussion, and concurrently make that change available via our
> astronomer images of 2.0 (which is available under Apache 2 License
> without commercial obligations, so usable by anyone in the community)
>
> Thanks for bringing this up.
>
> -ash
>
> On Sep 11 2020, at 12:56 pm, Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > I started to feel that we need to clarify statements about the HA
> Scheduler
> > for our community. Not that I am losing sleep regularly over this but it
> > did keep me away last night when I started to think about it :).
> >
> > I have a feeling that while we already defined some - rather
> > aggressive -
> > timelines for 2.0, the subject of HA Scheduler was not touched in the
> > previous Airflow 2.0 meetings. We are not very far from the release
> > but the
> > HA scheduler is implemented inside Astronomer and we have not seen any
> code
> > for it yet in the community. I understand that a lot of work (not only
> > development but especially testing) has been put into it from the
> > Astronomer team internally.
> >
> > I am actually quite OK with that to be like that. I think Astronomer
> > is a
> > super-valuable member of the community and I have no doubts Ash and Kaxil
> > and Daniel and others will do an awesome job with it. I am simply afraid
> > that when we see it, some of the cases that we see as needed by the
> > community will require more work. This will either delay the 2.0
> > release or
> > we will have to drop it from the 2.0 release. Looking at the number of
> > discussions we had with - much simpler IMHO - Smart Sensors, I have the
> > feeling that HA scheduler will spark even more discussions. The AIP-15
> > <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler
> >
> > was
> > not very rich in detail and the last time it has been updated was March
> > 2019 (!) and I have no doubt a big number of design decisions,
> > observations, learning has happened in Astronomer since.
> >
> > And to be perfectly honest - I am ok with both of the scenarios I can
> see:
> >
> > 1) We release HA Scheduler in 2.0
> >
> > For that, I think we should start looking at the code and discuss it
> > already quite some time ago IMHO. It might be too late if we want to fit
> > the aggressive timeline we have - especially that there are other things
> > the most active people are committing to for 2.0 and they might simply
> not
> > have enough time to make the quality review rounds and discussions. I
> think
> > we need to see it first to be even able to assess if we think we can make
> > it within the timeline.
> >
> > 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer
> will
> > use the HA Scheduler in their own service as a "commercial" add-on or
> > "advantage" of their offering.
> >
> > In the meantime - between 2.0 and 2.1 Astronomer could donate the code
> and
> > we could make sure it is reviewed and merged in the way that answers the
> > needs of different community members. This has also numerous
> > advantages to
> > the community - similar to the case of Smart Sensors, Astronomer can test
> > it in production then and solve all the teething problems of such a
> > service.
> >
> > I cannot speak for the business models of Astronomer of course :), but it
> > seems to me like a nice advantage to have for a while, from the business
> > point of view. And as a community, we also benefit that we have such a
> > strong member of the community with a sustainable and good business
> model.
> > Without Astronomer's generous support, Ash, Kaxil, and Daniel especially
> > (but also others) - Airflow would not be where it is today. And I
> > would be
> > 100% happy with such an approach as a PMC and member of the community
> > and I
> > support it a lot if Astronomer chooses this path.
> >
> > I think, however, it's the highest time that we decide and clearly
> > communicate it to the users as a community. At least I have a feeling
> that
> > without the community members, committers, and some heavy users being
> > involved in the open, and having time for quality review and discussion,
> > releasing HA in 2.0 might be not possible. And to just reiterate -
> > this has
> > nothing to do with the expected quality of the code and testing, but more
> > about potential differences in expectations, assumptions, understanding,
> > performance limitations, and anything else that might (and usually does)
> > come up.
> >
> > I think - since we already started to publish the schedule, this is the
> > right time that we make a decision on that and align expectations.
> >
> > Ry, Vikram - I'd love to hear what the intentions of Astronomer as a
> > company for the HA Scheduler are? I know as a group of committers we said
> > it a number of times that HA Scheduler will be in 2.0 so we built the
> > expectations among our users as a community. But maybe you really think
> > that pursuing scenario 2) (or maybe another scenario I have not thought
> > about) is the way to go for Astronomer?
> >
> > As I wrote above - I am personally perfectly fine with either of the
> > scenarios, and I think they are both beneficial for the community, but I
> > think we should discuss it, align expectations, and clearly
> > communicate as
> > the Apache Airflow community.
> >
> > J.
> >
> > --
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and
Astronomer/Community plans?
Posted by Ash Berlin-Taylor <as...@apache.org>.
Hi Jarek, and all.
You aren't the only one to have this thought -- it's been on my mind too.
Sadly I wasn't able to get the code in a PR-able state before heading
off on paternity leave. I have started separating out and submitting the
"easy"/preparatory PRs to try to lessen the size of the "main" PR:
https://github.com/apache/airflow/pull/10729
https://github.com/apache/airflow/pull/10710
https://github.com/apache/airflow/pull/10706
But yes, at some point it needs a large "big-bang" PR. That is coming.
We will have a discussion internally and see what course we think is
best for us to take. One possible path is we submit the PR to open
discussion, and concurrently make that change available via our
astronomer images of 2.0 (which is available under Apache 2 License
without commercial obligations, so usable by anyone in the community)
Thanks for bringing this up.
-ash
On Sep 11 2020, at 12:56 pm, Jarek Potiuk <Ja...@polidea.com> wrote:
> I started to feel that we need to clarify statements about the HA Scheduler
> for our community. Not that I am losing sleep regularly over this but it
> did keep me away last night when I started to think about it :).
>
> I have a feeling that while we already defined some - rather
> aggressive -
> timelines for 2.0, the subject of HA Scheduler was not touched in the
> previous Airflow 2.0 meetings. We are not very far from the release
> but the
> HA scheduler is implemented inside Astronomer and we have not seen any code
> for it yet in the community. I understand that a lot of work (not only
> development but especially testing) has been put into it from the
> Astronomer team internally.
>
> I am actually quite OK with that to be like that. I think Astronomer
> is a
> super-valuable member of the community and I have no doubts Ash and Kaxil
> and Daniel and others will do an awesome job with it. I am simply afraid
> that when we see it, some of the cases that we see as needed by the
> community will require more work. This will either delay the 2.0
> release or
> we will have to drop it from the 2.0 release. Looking at the number of
> discussions we had with - much simpler IMHO - Smart Sensors, I have the
> feeling that HA scheduler will spark even more discussions. The AIP-15
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-15+Scalable+Scheduler>
> was
> not very rich in detail and the last time it has been updated was March
> 2019 (!) and I have no doubt a big number of design decisions,
> observations, learning has happened in Astronomer since.
>
> And to be perfectly honest - I am ok with both of the scenarios I can see:
>
> 1) We release HA Scheduler in 2.0
>
> For that, I think we should start looking at the code and discuss it
> already quite some time ago IMHO. It might be too late if we want to fit
> the aggressive timeline we have - especially that there are other things
> the most active people are committing to for 2.0 and they might simply not
> have enough time to make the quality review rounds and discussions. I think
> we need to see it first to be even able to assess if we think we can make
> it within the timeline.
>
> 2) We agree to release the HA Scheduler in 2.1 (or 2.2) and Astronomer will
> use the HA Scheduler in their own service as a "commercial" add-on or
> "advantage" of their offering.
>
> In the meantime - between 2.0 and 2.1 Astronomer could donate the code and
> we could make sure it is reviewed and merged in the way that answers the
> needs of different community members. This has also numerous
> advantages to
> the community - similar to the case of Smart Sensors, Astronomer can test
> it in production then and solve all the teething problems of such a
> service.
>
> I cannot speak for the business models of Astronomer of course :), but it
> seems to me like a nice advantage to have for a while, from the business
> point of view. And as a community, we also benefit that we have such a
> strong member of the community with a sustainable and good business model.
> Without Astronomer's generous support, Ash, Kaxil, and Daniel especially
> (but also others) - Airflow would not be where it is today. And I
> would be
> 100% happy with such an approach as a PMC and member of the community
> and I
> support it a lot if Astronomer chooses this path.
>
> I think, however, it's the highest time that we decide and clearly
> communicate it to the users as a community. At least I have a feeling that
> without the community members, committers, and some heavy users being
> involved in the open, and having time for quality review and discussion,
> releasing HA in 2.0 might be not possible. And to just reiterate -
> this has
> nothing to do with the expected quality of the code and testing, but more
> about potential differences in expectations, assumptions, understanding,
> performance limitations, and anything else that might (and usually does)
> come up.
>
> I think - since we already started to publish the schedule, this is the
> right time that we make a decision on that and align expectations.
>
> Ry, Vikram - I'd love to hear what the intentions of Astronomer as a
> company for the HA Scheduler are? I know as a group of committers we said
> it a number of times that HA Scheduler will be in 2.0 so we built the
> expectations among our users as a community. But maybe you really think
> that pursuing scenario 2) (or maybe another scenario I have not thought
> about) is the way to go for Astronomer?
>
> As I wrote above - I am personally perfectly fine with either of the
> scenarios, and I think they are both beneficial for the community, but I
> think we should discuss it, align expectations, and clearly
> communicate as
> the Apache Airflow community.
>
> J.
>
> --
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>