You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@airflow.apache.org by Daniel Imberman <da...@gmail.com> on 2020/03/20 15:47:53 UTC

Let's talk Airflow 2.0

Hello, fellow Airflowers!

I wanted to start this discussion in hopes of "kicking the tires" and
agreeing on a timeline for the Airflow 2.0 release.

It seems to me that our 1.10 branch is straying farther and farther from
master, and with that, there are two significant consequences. Releases
become harder as we are rewriting PRs for the release branch, and we are
delaying releasing of already-merged features and upgrades.


But I want to add _____ feature which would be so awesome in a 2.0 press
release!

Our community has a lot of REALLY cool projects and efforts, and yes, it
would be awesome to have a shiny 2.0 release fully packed with amazing
features. Still, we are sacrificing two things with this mindset:
timeliness and stability. The longer we wait on releasing many of the
already added features of our master branch (e.g., a more stable typing
system, the ability to submit K8sExecutor pods as YAML files, etc.) the
more we risk falling behind to other projects in this field. We should also
consider that the 2.0 release is ALREADY going to take a monumental testing
effort, and adding more features will only make that worse.

With Apache Spark, some of the most significant recent features released
were the native Kubernetes integrations
<https://spark.apache.org/docs/2.3.0/running-on-kubernetes.html>. This
major paradigm shift was a crucial feature for Spark to remain competitive
in the data processing space, and the community released it in the
innocuous-sounding 2.3 release. Had they waited for spark 3.0, this feature
would still be unreleased.

The release process for 2.0 will take a lot of work/I'm not sure we have
time for it.

This reason is why it is ALL the more crucial that we start this process
now. This will only get harder, and we risk more bugs as master diverges
farther from 1.10.

So what should we do now?

Spring Cleaning of Necessary 2.0 tasks

A while back, we created a spring cleaning list of 2.0 necessary fixes.
Let's review, triage the tickets no longer relevant, and give the highest
priority to the remaining tickets.

Determine Breaking Changes

Of all of the significant efforts in play right now, let's determine which
of those efforts are breaking. A major version bump is the only time we can
reasonably expect users to accept modifying DAGs, so let's categorize those
issues and prioritize them.

Release API and Scheduler HA

Of the remaining efforts that could potentially cause breaking changes, I
believe that the new API, which replaces the “Experimental API”, and the
Scheduler HA have the most potential upside for users. Let's make sure to
include these in our planning for releasing 2.0.

Set a Timeline for 1.10 releases

One question users will ask when this release comes out is, "how long will
the 1.10.x release stream continue to be enhanced?" In creating a timeline,
we want to ensure that customers are not left high and dry while also
preventing any form of Python 2/3 issue where users feel no urgency to
update. I propose a 2 month window where the 1.10.x release stream will
continue getting critical bug fixes, but not any additional features.


Enhance Tests to Avoid Regressions

Let’s make sure that any new commits have tests suitable for both streams
while we get ready for 2.0. Let's continue our work on writing system
tests. Let’s all contribute scenarios so that we can ensure that these are
covered before we launch 2.0.


I'm ecstatic for the next year in Apache Airflow. Everything is speeding
up, and by cutting a 2.0 release, I think we can ensure that we keep up
with the pace of our growing community and demand.

Re: Let's talk Airflow 2.0

Posted by Daniel Imberman <da...@gmail.com>.

Hi Kaxil,

I’m currently working on creating the list of necessary fixes. One thing I’m prioritizing before we shift over is porting or deleting all of the old JIRA tickets. It would be great to start 2.0 with as clean of a slate as possible. However, I will try to get that proposal out sooner so others can review it in parallel with the ticket migration.

On Tue, Mar 31, 2020 at 11:08 AM, Kaxil Naik <ka...@gmail.com> wrote:
Did we get anywhere on this? What do others think?

For context: The code has moved quite a bit between Airflow 1.10 and
Airflow 2.0 and it becomes increasingly difficult to backport Core changes
without rewriting the PRs.

Regards,
Kaxil

On Tue, Mar 24, 2020 at 2:54 PM Daniel Imberman <da...@gmail.com>
wrote:

> Hi Robin,
>
> > > I feel some of the stuff for instance Schedular HA could wait for a
> point
> > > release of version 2 (although maybe this a lot further a long than I
> am
> > > aware). Like you mentioned Spark did with K8s.
> > >
> I agree on this part. The focus would be more on the breaking changes and
> major features. @ash has been determining which features can wait.
>
> > > Also does the new API need to be feature complete or just enough
> > > functionality to warrant removing the existing experimental one.
>
> It needs to be complete enough for us to run full integration tests and
> for us to assume that we will not need major revisions in the near future.
> It will of course continue to evolve over time (though hopefully mostly
> through additions).
>
> Daniel
> On Mar 24, 2020, 6:33 AM -0700, Kaxil Naik <ka...@gmail.com>, wrote:
> > +1
> >
> >
> >
> > On Sun, Mar 22, 2020 at 8:19 AM Robin Edwards <ro...@bidnamic.com> wrote:
> >
> > > I feel some of the stuff for instance Schedular HA could wait for a
> point
> > > release of version 2 (although maybe this a lot further a long than I
> am
> > > aware). Like you mentioned Spark did with K8s.
> > >
> > > Also does the new API need to be feature complete or just enough
> > > functionality to warrant removing the existing experimental one.
> > >
> > > Like you said, less sooner is better.
> > >
> > > R
> > >
> > > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <daniel.imberman@gmail.com
> >
> > > wrote:
> > >
> > > > Great! Hope to get a few more folx to give +1's but I think we have
> a good
> > > > path forward here :)
> > > >
> > > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > > wrote:
> > > >
> > > > > >
> > > > > >
> > > > > > I agree especially for larger-scale users migrations are a
> difficult
> > > > > > process. Perhaps we can adopt something similar to a blockchain
> fork
> > > > > (e.g.
> > > > > > determine X known airflow using companies, and start the
> countdown as
> > > > > soon
> > > > > > as Y% of them migrate). I really just want to make sure we don't
> end
> > > > up
> > > > > > with a python2/3 situation. Even if we continue support it should
> > > > only be
> > > > > > for bugfixes and we should not add any new features into 1.10.
> > > > > >
> > > > >
> > > > > I think we are in perfect sync - I think feature-migration should
> end
> > > > > almost immediately after we release 2.0. But bug-fixing should
> continue
> > > > > for quite some time. On that front - having backport packages will
> help
> > > > > with releasing "integrations" quite independently from 1.10/2.0
> version
> > > > > (which I think is good for those who are - for this or another
> reason -
> > > > > stuck on 2.0). On the other hand we should make sure that the
> important
> > > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For
> > > > example
> > > > > a lot of recent performance improvements that we have now in 2.0
> will
> > > > > be possible (and not that complex) to backport to 1.10. Some of
> this
> > > > effort
> > > > > is actually easier to do in 2.0 and then apply to 1.10 in similar
> > > > fashion
> > > > > as it is easier to understand and reason about the 2.0 code now
> when
> > > > > we have some refactoring/pylints etc in place. So we should make
> sure
> > > > > we get the latest 1.10 to a "good" state - before we freeze it for
> > > > bugfix
> > > > > only.
> > > > > I know it might mean that some people will stay with 1.10 for
> longer,
> > > > but
> > > > > that's also OK for them. The reason to migrate to 2.0 should be not
> > > > > performance but some important features (like API or HA) that come
> > > > > with it.
> > > > >
> > > > > I couldn't agree more :). If we can start people writing (close
> to) 2.0
> > > > > > compliant DAGs before the release of 2.0 that will make the
> migration
> > > > > > process so much easier :).
> > > > > >
> > > > >
> > > > > Yeah. I even thought that we should write a
> > > > > "How good your DAGs are for 2.0" assessment tool.
> > > > >
> > > > >
> > > > > > If there aren't any extra steps or features that we need to add
> > > > (beyond
> > > > > the
> > > > > > ones discussed here), I think a good next step would be to
> create an
> > > > > > official checklist just so we can see all of these features in
> one
> > > > place
> > > > > > (and hopefully start breaking them down into as small of changes
> as
> > > > > > possible).
> > > > > >
> > > > > > Does that sound ok?
> > > > > >
> > > > >
> > > > > Perfectly OK for me!
> > > > >
> > > > >
> > > > > >
> > > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129 <+48660796129>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > > > >
> > > >
> > >
>

Re: Let's talk Airflow 2.0

Posted by Kaxil Naik <ka...@gmail.com>.

Did we get anywhere on this? What do others think?

For context: The code has moved quite a bit between Airflow 1.10 and
Airflow 2.0 and it becomes increasingly difficult to backport Core changes
without rewriting the PRs.

Regards,
Kaxil

On Tue, Mar 24, 2020 at 2:54 PM Daniel Imberman <da...@gmail.com>
wrote:

> Hi Robin,
>
> > > I feel some of the stuff for instance Schedular HA could wait for a
> point
> > > release of version 2 (although maybe this a lot further a long than I
> am
> > > aware). Like you mentioned Spark did with K8s.
> > >
> I agree on this part. The focus would be more on the breaking changes and
> major features. @ash has been determining which features can wait.
>
> > > Also does the new API need to be feature complete or just enough
> > > functionality to warrant removing the existing experimental one.
>
> It needs to be complete enough for us to run full integration tests and
> for us to assume that we will not need major revisions in the near future.
> It will of course continue to evolve over time (though hopefully mostly
> through additions).
>
> Daniel
> On Mar 24, 2020, 6:33 AM -0700, Kaxil Naik <ka...@gmail.com>, wrote:
> > +1
> >
> >
> >
> > On Sun, Mar 22, 2020 at 8:19 AM Robin Edwards <ro...@bidnamic.com> wrote:
> >
> > > I feel some of the stuff for instance Schedular HA could wait for a
> point
> > > release of version 2 (although maybe this a lot further a long than I
> am
> > > aware). Like you mentioned Spark did with K8s.
> > >
> > > Also does the new API need to be feature complete or just enough
> > > functionality to warrant removing the existing experimental one.
> > >
> > > Like you said, less sooner is better.
> > >
> > > R
> > >
> > > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <daniel.imberman@gmail.com
> >
> > > wrote:
> > >
> > > > Great! Hope to get a few more folx to give +1's but I think we have
> a good
> > > > path forward here :)
> > > >
> > > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > > wrote:
> > > >
> > > > > >
> > > > > >
> > > > > > I agree especially for larger-scale users migrations are a
> difficult
> > > > > > process. Perhaps we can adopt something similar to a blockchain
> fork
> > > > > (e.g.
> > > > > > determine X known airflow using companies, and start the
> countdown as
> > > > > soon
> > > > > > as Y% of them migrate). I really just want to make sure we don't
> end
> > > > up
> > > > > > with a python2/3 situation. Even if we continue support it should
> > > > only be
> > > > > > for bugfixes and we should not add any new features into 1.10.
> > > > > >
> > > > >
> > > > > I think we are in perfect sync - I think feature-migration should
> end
> > > > > almost immediately after we release 2.0. But bug-fixing should
> continue
> > > > > for quite some time. On that front - having backport packages will
> help
> > > > > with releasing "integrations" quite independently from 1.10/2.0
> version
> > > > > (which I think is good for those who are - for this or another
> reason -
> > > > > stuck on 2.0). On the other hand we should make sure that the
> important
> > > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For
> > > > example
> > > > > a lot of recent performance improvements that we have now in 2.0
> will
> > > > > be possible (and not that complex) to backport to 1.10. Some of
> this
> > > > effort
> > > > > is actually easier to do in 2.0 and then apply to 1.10 in similar
> > > > fashion
> > > > > as it is easier to understand and reason about the 2.0 code now
> when
> > > > > we have some refactoring/pylints etc in place. So we should make
> sure
> > > > > we get the latest 1.10 to a "good" state - before we freeze it for
> > > > bugfix
> > > > > only.
> > > > > I know it might mean that some people will stay with 1.10 for
> longer,
> > > > but
> > > > > that's also OK for them. The reason to migrate to 2.0 should be not
> > > > > performance but some important features (like API or HA) that come
> > > > > with it.
> > > > >
> > > > > I couldn't agree more :). If we can start people writing (close
> to) 2.0
> > > > > > compliant DAGs before the release of 2.0 that will make the
> migration
> > > > > > process so much easier :).
> > > > > >
> > > > >
> > > > > Yeah. I even thought that we should write a
> > > > > "How good your DAGs are for 2.0" assessment tool.
> > > > >
> > > > >
> > > > > > If there aren't any extra steps or features that we need to add
> > > > (beyond
> > > > > the
> > > > > > ones discussed here), I think a good next step would be to
> create an
> > > > > > official checklist just so we can see all of these features in
> one
> > > > place
> > > > > > (and hopefully start breaking them down into as small of changes
> as
> > > > > > possible).
> > > > > >
> > > > > > Does that sound ok?
> > > > > >
> > > > >
> > > > > Perfectly OK for me!
> > > > >
> > > > >
> > > > > >
> > > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129 <+48660796129>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > > > >
> > > >
> > >
>

Re: Let's talk Airflow 2.0

Posted by Daniel Imberman <da...@gmail.com>.

Hi Robin,

> > I feel some of the stuff for instance Schedular HA could wait for a point
> > release of version 2 (although maybe this a lot further a long than I am
> > aware). Like you mentioned Spark did with K8s.
> >
I agree on this part. The focus would be more on the breaking changes and major features. @ash has been determining which features can wait.

> > Also does the new API need to be feature complete or just enough
> > functionality to warrant removing the existing experimental one.

It needs to be complete enough for us to run full integration tests and for us to assume that we will not need major revisions in the near future. It will of course continue to evolve over time (though hopefully mostly through additions).

Daniel
On Mar 24, 2020, 6:33 AM -0700, Kaxil Naik <ka...@gmail.com>, wrote:
> +1
>
>
>
> On Sun, Mar 22, 2020 at 8:19 AM Robin Edwards <ro...@bidnamic.com> wrote:
>
> > I feel some of the stuff for instance Schedular HA could wait for a point
> > release of version 2 (although maybe this a lot further a long than I am
> > aware). Like you mentioned Spark did with K8s.
> >
> > Also does the new API need to be feature complete or just enough
> > functionality to warrant removing the existing experimental one.
> >
> > Like you said, less sooner is better.
> >
> > R
> >
> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <da...@gmail.com>
> > wrote:
> >
> > > Great! Hope to get a few more folx to give +1's but I think we have a good
> > > path forward here :)
> > >
> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <Ja...@polidea.com>
> > > wrote:
> > >
> > > > >
> > > > >
> > > > > I agree especially for larger-scale users migrations are a difficult
> > > > > process. Perhaps we can adopt something similar to a blockchain fork
> > > > (e.g.
> > > > > determine X known airflow using companies, and start the countdown as
> > > > soon
> > > > > as Y% of them migrate). I really just want to make sure we don't end
> > > up
> > > > > with a python2/3 situation. Even if we continue support it should
> > > only be
> > > > > for bugfixes and we should not add any new features into 1.10.
> > > > >
> > > >
> > > > I think we are in perfect sync - I think feature-migration should end
> > > > almost immediately after we release 2.0. But bug-fixing should continue
> > > > for quite some time. On that front - having backport packages will help
> > > > with releasing "integrations" quite independently from 1.10/2.0 version
> > > > (which I think is good for those who are - for this or another reason -
> > > > stuck on 2.0). On the other hand we should make sure that the important
> > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For
> > > example
> > > > a lot of recent performance improvements that we have now in 2.0 will
> > > > be possible (and not that complex) to backport to 1.10. Some of this
> > > effort
> > > > is actually easier to do in 2.0 and then apply to 1.10 in similar
> > > fashion
> > > > as it is easier to understand and reason about the 2.0 code now when
> > > > we have some refactoring/pylints etc in place. So we should make sure
> > > > we get the latest 1.10 to a "good" state - before we freeze it for
> > > bugfix
> > > > only.
> > > > I know it might mean that some people will stay with 1.10 for longer,
> > > but
> > > > that's also OK for them. The reason to migrate to 2.0 should be not
> > > > performance but some important features (like API or HA) that come
> > > > with it.
> > > >
> > > > I couldn't agree more :). If we can start people writing (close to) 2.0
> > > > > compliant DAGs before the release of 2.0 that will make the migration
> > > > > process so much easier :).
> > > > >
> > > >
> > > > Yeah. I even thought that we should write a
> > > > "How good your DAGs are for 2.0" assessment tool.
> > > >
> > > >
> > > > > If there aren't any extra steps or features that we need to add
> > > (beyond
> > > > the
> > > > > ones discussed here), I think a good next step would be to create an
> > > > > official checklist just so we can see all of these features in one
> > > place
> > > > > (and hopefully start breaking them down into as small of changes as
> > > > > possible).
> > > > >
> > > > > Does that sound ok?
> > > > >
> > > >
> > > > Perfectly OK for me!
> > > >
> > > >
> > > > >
> > > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
> >

Re: Let's talk Airflow 2.0

Posted by Kaxil Naik <ka...@gmail.com>.

+1



On Sun, Mar 22, 2020 at 8:19 AM Robin Edwards <ro...@bidnamic.com> wrote:

> I feel some of the stuff for instance Schedular HA could wait for a point
> release of version 2  (although maybe this a lot further a long than I am
> aware). Like you mentioned Spark did with K8s.
>
> Also does the new API need to be feature complete or just enough
> functionality to warrant removing the existing experimental one.
>
> Like you said, less sooner is better.
>
> R
>
> On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <da...@gmail.com>
> wrote:
>
>> Great! Hope to get a few more folx to give +1's but I think we have a good
>> path forward here :)
>>
>> On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> > >
>> > >
>> > > I agree especially for larger-scale users migrations are a difficult
>> > > process. Perhaps we can adopt something similar to a blockchain fork
>> > (e.g.
>> > > determine X known airflow using companies, and start the countdown as
>> > soon
>> > > as Y% of them migrate). I really just want to make sure we don't end
>> up
>> > > with a python2/3 situation. Even if we continue support it should
>> only be
>> > > for bugfixes and we should not add any new features into 1.10.
>> > >
>> >
>> > I think we are in perfect sync  - I think feature-migration should end
>> > almost immediately after we release 2.0. But bug-fixing should continue
>> > for quite some time. On that front - having backport packages will help
>> > with releasing "integrations" quite independently from 1.10/2.0 version
>> > (which I think is good for those who are - for this or another reason -
>> > stuck on 2.0). On the other hand we should make sure that the important
>> > stuff for 2.0 that is not "feature" is also backported to 1.10. For
>> example
>> > a lot of recent performance improvements that we have now in 2.0 will
>> > be possible (and not that complex) to backport to 1.10. Some of this
>> effort
>> > is actually easier to do in 2.0 and then apply to 1.10 in similar
>> fashion
>> > as it is easier to understand and reason about the 2.0 code now when
>> > we have some refactoring/pylints etc in place. So we should make sure
>> > we get the latest 1.10 to a "good" state - before we freeze it for
>> bugfix
>> > only.
>> > I know it might mean that some people will stay with 1.10 for longer,
>> but
>> > that's also OK for them. The reason to migrate to 2.0 should be not
>> > performance but some important features (like API or HA) that come
>> > with it.
>> >
>> > I couldn't agree more :). If we can start people writing (close to) 2.0
>> > > compliant DAGs before the release of 2.0 that will make the migration
>> > > process so much easier :).
>> > >
>> >
>> > Yeah.  I even thought that we should write a
>> > "How good your DAGs are for 2.0" assessment tool.
>> >
>> >
>> > > If there aren't any extra steps or features that we need to add
>> (beyond
>> > the
>> > > ones discussed here), I think a good next step would be to create an
>> > > official checklist just so we can see all of these features in one
>> place
>> > > (and hopefully start breaking them down into as small of changes as
>> > > possible).
>> > >
>> > > Does that sound ok?
>> > >
>> >
>> > Perfectly OK for me!
>> >
>> >
>> > >
>> > > --
>> >
>> > Jarek Potiuk
>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >
>> > M: +48 660 796 129 <+48660796129>
>> > [image: Polidea] <https://www.polidea.com/>
>> >
>>
>

Re: Let's talk Airflow 2.0

Posted by Kamil Breguła <ka...@polidea.com>.

4. Improve Scheduler performance and reliability

The performance was improved by my work.
https://lists.apache.org/thread.html/r4b0a6385d263defaf0fe733a48fb291e3a763da2f172ca1931a652b7%40%3Cdev.airflow.apache.org%3E

In Q2, my team will continue to work on performance and reliability.
We want to prepare tests that will detect potential problems and fix
them.

The reliability will be improved by HA. Ash working on it. It is
covered by AIP-15
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651

5. Extend/finish the API

Voting is open. There is one topic for discussion, but I hope we can
find a consensus soon.
https://lists.apache.org/thread.html/rcc379dc7067397e1a631c19b9f194f2f572a2ea7a338648788911c91%40%3Cdev.airflow.apache.org%3E
This is the most important area I will want to work on right now.

Re: Let's talk Airflow 2.0

Posted by Jarek Potiuk <Ja...@polidea.com>.

6. Prod image is there :) -> alpha quality for now and ready to be tested.
And I have plans to add more tests and with Daniel/Greg integrate it with
the helm chart and make both - the image and helm the "official" ones.
Alpha version of the image will also be released with 1.10.10 so that
people can start testing it before 2.0 (it's pretty much the same image).

Worth mentioning is that we have the backport packages (in progress). We
have just finished an important testing milestone for those (whole
'airflow-providers-google' package is fully system-tested with 1.10.9 and
1.10.6). I am going to pick it up for the official release process and
other packages next week.  We will start working on system test automation
(so hopefully tests for 1.10.10 will run fully automatically on Github
Actions + GCP - paving the way for other packages).

The Backport packages are - I think - an important step to ease the 2.0
migration as they will let people start testing the operators/hooks/sensors
from 2.0 before an actual release and likely we will be able to live-test
big chunk of 2.0 release before it is actually released and it will later
(beyond 2.0) let us implement AIP-8 - splitting Airflow to core/providers.

J.

On Fri, Apr 3, 2020 at 9:38 AM Kamil Breguła <ka...@polidea.com>
wrote:

> >
> > Move (tested) components out of contrib folder
> >
> >
> https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E
>
>
> AIP-21 is done. We have a follow-up in the form of system tests and
> backport packages, but this is not a blocker. We can go further without
> system tests.
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths
>
>
> On Thu, Apr 2, 2020 at 6:55 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
> > Hello all,
> >
> > I've been reviewing This wiki page
> > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0> and I
> > wanted to discuss which of these features are still relevant/are blockers
> > for 2.0
> >
> > I figured we should start with the status of the high priority fixes
> >
> >
> >    1.
> > *Knative Executor (handled by KEDA) *The KEDA Autoscaler has accomplished
> >    everything that we would've hoped for from a KnativeExecutor. I think
> we
> >    can consider this work done.
> >    2. *Improve Webserver performance*
> >    @Kaxil Naik <ka...@gmail.com> It appears that DAG serialization
> is
> >    already merged? Are there other steps needed for webserver
> performance?
> >    3.
> > *Enhanced real-time UI *While this would make for a great "wow!" feature,
> >    I don't think this is necessary for a 2.0. Unless anyone wants to
> work on
> >    this in the short term :).
> >    4.
> > *Improve Scheduler performance and reliability @Ash Berlin-Taylor
> >    <as...@apache.org> *has been hard at work on scheduler HA, and we've
> >    already
> >    5.
> > *Extend/finish the API @Kamil Breguła <ka...@polidea.com> *I see
> >    that there is an active PR here
> >    <https://github.com/apache/airflow/pull/7549>, are we getting close
> on
> >    this?
> >    6.
> > *Production Docker image *The PR here should be merged hopefully today!
> >
> > Now there's a second section of "needs more info" tickets and I wanted to
> > see which of these we could triage for after 2.0
> >
> > Rework Subdags to be less "bolted-on" and more native to the scheduler
> >>
> >> There are all sorts of edge cases around subdags that result from the
> >> tasks in the subdag being run by another executor, instead of being
> handled
> >> and scheduled by the core Scheduler. We should make the scheduler "see
> in"
> >> to the Subdags and make it responsible for scheduling tasks. This should
> >> make subdags less error prone and more predictable. It may involve
> >> replacing/significantly changing the SubDagOperator
> >
> >
> > I think it's pretty damn crucial we fix subdags and backfills. I'm on the
> > fence about this one. On the one hand it could possibly wait. On the
> other
> > hand it would be embarrasing to release a 2.0 and still have this feature
> > broken
> >
> > Move (tested) components out of contrib folder
> >>
> >>
> https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E
> >>
> >
> > I think this has been something we've actively worked on. Is this task
> > complete?
> >
> > Filter passwords/sensitive info from logs.
> >>
> >> Jenkins does this if the password comes from a connection - it would be
> >> good if we could do this too
> >
> >
> > While I agree that this is an important issue, I think if no one has
> > picked this up yet we should triage for a later release (I also don't
> think
> > this should break anything)
> >
> > Allow Backfill runs to be handled by the scheduler/triggered from UI
> >>
> >> It would be nice to not need console access to run airflow backfill, and
> >> to have not it not stop if the SSH session is closed.
> >>
> >> Lots of details to work out here though around how this would work,
> where
> >> would it show up in UI, priority of tasks, ways of reducing
> >> concurrency/load to allow normal tasks to run etc.
> >
> >
> > Once again I think this is a feature that's a bit embarrasing NOT to fix
> > for a 2.0 release, but also understand that there's only so much
> manpower.
> > I also imagine this would be a HUGE undertaking so I think this would
> push
> > back 2.0 significantly.
> >
> >
> > Rationalize HA in Connections
> >>
> >> Right now it is possible to create multiple connections with the same ID
> >> and *some*  Connections/hooks will support this and pick a random one
> >> from the list. This feature isn't well documented or understood (and the
> >> CLI doesn't support it as well as the UI for instance) so we should
> examine
> >> if this makes sense, or if we should support it individually in certain
> >> connection types instead.
> >
> >
> > I honestly don't know enough about this to have a strong opinion on it
> >
> > Make setting up HTTPS connections easier/more expected
> >>
> >> It appears that @Kamil Breguła <ka...@polidea.com>  has an open
> > PR <https://github.com/apache/airflow/pull/5239/files> for this one.
> @Kamil
> > Breguła <ka...@polidea.com> do you think this would be hard to
> > merge?
> >
> > Front end/"browser" testing
> >>
> >> The Airflow UI is non trivial and there have been a number of JS/html
> >> bugs that could have been caught by better front-end testing.
> >>
> >> It has been suggested to look at Cypress for this over Selenium. What
> >> ever we choose we need to pay careful attention to avoid slow or flakey
> UI
> >> tests.
> >>
> >
> > This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've
> > been taking time to learn cypress recently, and once the airflow helm
> chart
> > is merged I think merging a set of integration/behavior/UI tests is
> crucial
> >
> > What does everyone think? Open to suggestions on this assessment!
> >
> >
> > On Tue, Mar 31, 2020 at 11:46 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> >> Got it. Thanks Daniel for leading this
> >>
> >> On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman <
> >> daniel.imberman@gmail.com> wrote:
> >>
> >>> I think including both is fine as long as the old one contains
> >>> deprecation warnings/force a feature flag to allow it
> >>> (e.g. —allow-deprecated)
> >>>
> >>> via Newton Mail
> >>> <
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> >
> >>>
> >>> On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła <
> >>> kamil.bregula@polidea.com> wrote:
> >>>
> >>> On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com>
> wrote:
> >>>
> >>> > Also does the new API need to be feature complete or just enough
> >>> > functionality to warrant removing the existing experimental one.
> >>> >
> >>>
> >>> I think we should release at least one version that will contain the
> >>> new and old REST APIs simultaneously. It is not easy to upgrade two
> >>> complex systems at the same time. However, if we do this, some users
> >>> will have to do it. Older versions can be hidden behind the feature
> >>> gate. We can also add deprecation warnings.
> >>>
> >>> >
> >>> > R
> >>> >
> >>> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <
> daniel.imberman@gmail.com
> >>> >
> >>> > wrote:
> >>> >
> >>> > > Great! Hope to get a few more folx to give +1's but I think we have
> >>> a good
> >>> > > path forward here :)
> >>> > >
> >>> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
> >>> Jarek.Potiuk@polidea.com>
> >>> > > wrote:
> >>> > >
> >>> > > > >
> >>> > > > >
> >>> > > > > I agree especially for larger-scale users migrations are a
> >>> difficult
> >>> > > > > process. Perhaps we can adopt something similar to a blockchain
> >>> fork
> >>> > > > (e.g.
> >>> > > > > determine X known airflow using companies, and start the
> >>> countdown as
> >>> > > > soon
> >>> > > > > as Y% of them migrate). I really just want to make sure we
> don't
> >>> end up
> >>> > > > > with a python2/3 situation. Even if we continue support it
> >>> should only
> >>> > > be
> >>> > > > > for bugfixes and we should not add any new features into 1.10.
> >>> > > > >
> >>> > > >
> >>> > > > I think we are in perfect sync - I think feature-migration should
> >>> end
> >>> > > > almost immediately after we release 2.0. But bug-fixing should
> >>> continue
> >>> > > > for quite some time. On that front - having backport packages
> will
> >>> help
> >>> > > > with releasing "integrations" quite independently from 1.10/2.0
> >>> version
> >>> > > > (which I think is good for those who are - for this or another
> >>> reason -
> >>> > > > stuck on 2.0). On the other hand we should make sure that the
> >>> important
> >>> > > > stuff for 2.0 that is not "feature" is also backported to 1.10.
> For
> >>> > > example
> >>> > > > a lot of recent performance improvements that we have now in 2.0
> >>> will
> >>> > > > be possible (and not that complex) to backport to 1.10. Some of
> >>> this
> >>> > > effort
> >>> > > > is actually easier to do in 2.0 and then apply to 1.10 in similar
> >>> fashion
> >>> > > > as it is easier to understand and reason about the 2.0 code now
> >>> when
> >>> > > > we have some refactoring/pylints etc in place. So we should make
> >>> sure
> >>> > > > we get the latest 1.10 to a "good" state - before we freeze it
> for
> >>> bugfix
> >>> > > > only.
> >>> > > > I know it might mean that some people will stay with 1.10 for
> >>> longer, but
> >>> > > > that's also OK for them. The reason to migrate to 2.0 should be
> not
> >>> > > > performance but some important features (like API or HA) that
> come
> >>> > > > with it.
> >>> > > >
> >>> > > > I couldn't agree more :). If we can start people writing (close
> >>> to) 2.0
> >>> > > > > compliant DAGs before the release of 2.0 that will make the
> >>> migration
> >>> > > > > process so much easier :).
> >>> > > > >
> >>> > > >
> >>> > > > Yeah. I even thought that we should write a
> >>> > > > "How good your DAGs are for 2.0" assessment tool.
> >>> > > >
> >>> > > >
> >>> > > > > If there aren't any extra steps or features that we need to add
> >>> (beyond
> >>> > > > the
> >>> > > > > ones discussed here), I think a good next step would be to
> >>> create an
> >>> > > > > official checklist just so we can see all of these features in
> >>> one
> >>> > > place
> >>> > > > > (and hopefully start breaking them down into as small of
> changes
> >>> as
> >>> > > > > possible).
> >>> > > > >
> >>> > > > > Does that sound ok?
> >>> > > > >
> >>> > > >
> >>> > > > Perfectly OK for me!
> >>> > > >
> >>> > > >
> >>> > > > >
> >>> > > > > --
> >>> > > >
> >>> > > > Jarek Potiuk
> >>> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>> > > >
> >>> > > > M: +48 660 796 129 <+48660796129>
> >>> > > > [image: Polidea] <https://www.polidea.com/>
> >>> > > >
> >>> > >
> >>>
> >>>
>


-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Re: Let's talk Airflow 2.0

Posted by Kamil Breguła <ka...@polidea.com>.

>
> Move (tested) components out of contrib folder
>
> https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E


AIP-21 is done. We have a follow-up in the form of system tests and
backport packages, but this is not a blocker. We can go further without
system tests.
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths


On Thu, Apr 2, 2020 at 6:55 PM Daniel Imberman <da...@gmail.com>
wrote:

> Hello all,
>
> I've been reviewing This wiki page
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0> and I
> wanted to discuss which of these features are still relevant/are blockers
> for 2.0
>
> I figured we should start with the status of the high priority fixes
>
>
>    1.
> *Knative Executor (handled by KEDA) *The KEDA Autoscaler has accomplished
>    everything that we would've hoped for from a KnativeExecutor. I think we
>    can consider this work done.
>    2. *Improve Webserver performance*
>    @Kaxil Naik <ka...@gmail.com> It appears that DAG serialization is
>    already merged? Are there other steps needed for webserver performance?
>    3.
> *Enhanced real-time UI *While this would make for a great "wow!" feature,
>    I don't think this is necessary for a 2.0. Unless anyone wants to work on
>    this in the short term :).
>    4.
> *Improve Scheduler performance and reliability @Ash Berlin-Taylor
>    <as...@apache.org> *has been hard at work on scheduler HA, and we've
>    already
>    5.
> *Extend/finish the API @Kamil Breguła <ka...@polidea.com> *I see
>    that there is an active PR here
>    <https://github.com/apache/airflow/pull/7549>, are we getting close on
>    this?
>    6.
> *Production Docker image *The PR here should be merged hopefully today!
>
> Now there's a second section of "needs more info" tickets and I wanted to
> see which of these we could triage for after 2.0
>
> Rework Subdags to be less "bolted-on" and more native to the scheduler
>>
>> There are all sorts of edge cases around subdags that result from the
>> tasks in the subdag being run by another executor, instead of being handled
>> and scheduled by the core Scheduler. We should make the scheduler "see in"
>> to the Subdags and make it responsible for scheduling tasks. This should
>> make subdags less error prone and more predictable. It may involve
>> replacing/significantly changing the SubDagOperator
>
>
> I think it's pretty damn crucial we fix subdags and backfills. I'm on the
> fence about this one. On the one hand it could possibly wait. On the other
> hand it would be embarrasing to release a 2.0 and still have this feature
> broken
>
> Move (tested) components out of contrib folder
>>
>> https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E
>>
>
> I think this has been something we've actively worked on. Is this task
> complete?
>
> Filter passwords/sensitive info from logs.
>>
>> Jenkins does this if the password comes from a connection - it would be
>> good if we could do this too
>
>
> While I agree that this is an important issue, I think if no one has
> picked this up yet we should triage for a later release (I also don't think
> this should break anything)
>
> Allow Backfill runs to be handled by the scheduler/triggered from UI
>>
>> It would be nice to not need console access to run airflow backfill, and
>> to have not it not stop if the SSH session is closed.
>>
>> Lots of details to work out here though around how this would work, where
>> would it show up in UI, priority of tasks, ways of reducing
>> concurrency/load to allow normal tasks to run etc.
>
>
> Once again I think this is a feature that's a bit embarrasing NOT to fix
> for a 2.0 release, but also understand that there's only so much manpower.
> I also imagine this would be a HUGE undertaking so I think this would push
> back 2.0 significantly.
>
>
> Rationalize HA in Connections
>>
>> Right now it is possible to create multiple connections with the same ID
>> and *some*  Connections/hooks will support this and pick a random one
>> from the list. This feature isn't well documented or understood (and the
>> CLI doesn't support it as well as the UI for instance) so we should examine
>> if this makes sense, or if we should support it individually in certain
>> connection types instead.
>
>
> I honestly don't know enough about this to have a strong opinion on it
>
> Make setting up HTTPS connections easier/more expected
>>
>> It appears that @Kamil Breguła <ka...@polidea.com>  has an open
> PR <https://github.com/apache/airflow/pull/5239/files> for this one. @Kamil
> Breguła <ka...@polidea.com> do you think this would be hard to
> merge?
>
> Front end/"browser" testing
>>
>> The Airflow UI is non trivial and there have been a number of JS/html
>> bugs that could have been caught by better front-end testing.
>>
>> It has been suggested to look at Cypress for this over Selenium. What
>> ever we choose we need to pay careful attention to avoid slow or flakey UI
>> tests.
>>
>
> This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've
> been taking time to learn cypress recently, and once the airflow helm chart
> is merged I think merging a set of integration/behavior/UI tests is crucial
>
> What does everyone think? Open to suggestions on this assessment!
>
>
> On Tue, Mar 31, 2020 at 11:46 AM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Got it. Thanks Daniel for leading this
>>
>> On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman <
>> daniel.imberman@gmail.com> wrote:
>>
>>> I think including both is fine as long as the old one contains
>>> deprecation warnings/force a feature flag to allow it
>>> (e.g. —allow-deprecated)
>>>
>>> via Newton Mail
>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2>
>>>
>>> On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła <
>>> kamil.bregula@polidea.com> wrote:
>>>
>>> On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com> wrote:
>>>
>>> > Also does the new API need to be feature complete or just enough
>>> > functionality to warrant removing the existing experimental one.
>>> >
>>>
>>> I think we should release at least one version that will contain the
>>> new and old REST APIs simultaneously. It is not easy to upgrade two
>>> complex systems at the same time. However, if we do this, some users
>>> will have to do it. Older versions can be hidden behind the feature
>>> gate. We can also add deprecation warnings.
>>>
>>> >
>>> > R
>>> >
>>> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <daniel.imberman@gmail.com
>>> >
>>> > wrote:
>>> >
>>> > > Great! Hope to get a few more folx to give +1's but I think we have
>>> a good
>>> > > path forward here :)
>>> > >
>>> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com>
>>> > > wrote:
>>> > >
>>> > > > >
>>> > > > >
>>> > > > > I agree especially for larger-scale users migrations are a
>>> difficult
>>> > > > > process. Perhaps we can adopt something similar to a blockchain
>>> fork
>>> > > > (e.g.
>>> > > > > determine X known airflow using companies, and start the
>>> countdown as
>>> > > > soon
>>> > > > > as Y% of them migrate). I really just want to make sure we don't
>>> end up
>>> > > > > with a python2/3 situation. Even if we continue support it
>>> should only
>>> > > be
>>> > > > > for bugfixes and we should not add any new features into 1.10.
>>> > > > >
>>> > > >
>>> > > > I think we are in perfect sync - I think feature-migration should
>>> end
>>> > > > almost immediately after we release 2.0. But bug-fixing should
>>> continue
>>> > > > for quite some time. On that front - having backport packages will
>>> help
>>> > > > with releasing "integrations" quite independently from 1.10/2.0
>>> version
>>> > > > (which I think is good for those who are - for this or another
>>> reason -
>>> > > > stuck on 2.0). On the other hand we should make sure that the
>>> important
>>> > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For
>>> > > example
>>> > > > a lot of recent performance improvements that we have now in 2.0
>>> will
>>> > > > be possible (and not that complex) to backport to 1.10. Some of
>>> this
>>> > > effort
>>> > > > is actually easier to do in 2.0 and then apply to 1.10 in similar
>>> fashion
>>> > > > as it is easier to understand and reason about the 2.0 code now
>>> when
>>> > > > we have some refactoring/pylints etc in place. So we should make
>>> sure
>>> > > > we get the latest 1.10 to a "good" state - before we freeze it for
>>> bugfix
>>> > > > only.
>>> > > > I know it might mean that some people will stay with 1.10 for
>>> longer, but
>>> > > > that's also OK for them. The reason to migrate to 2.0 should be not
>>> > > > performance but some important features (like API or HA) that come
>>> > > > with it.
>>> > > >
>>> > > > I couldn't agree more :). If we can start people writing (close
>>> to) 2.0
>>> > > > > compliant DAGs before the release of 2.0 that will make the
>>> migration
>>> > > > > process so much easier :).
>>> > > > >
>>> > > >
>>> > > > Yeah. I even thought that we should write a
>>> > > > "How good your DAGs are for 2.0" assessment tool.
>>> > > >
>>> > > >
>>> > > > > If there aren't any extra steps or features that we need to add
>>> (beyond
>>> > > > the
>>> > > > > ones discussed here), I think a good next step would be to
>>> create an
>>> > > > > official checklist just so we can see all of these features in
>>> one
>>> > > place
>>> > > > > (and hopefully start breaking them down into as small of changes
>>> as
>>> > > > > possible).
>>> > > > >
>>> > > > > Does that sound ok?
>>> > > > >
>>> > > >
>>> > > > Perfectly OK for me!
>>> > > >
>>> > > >
>>> > > > >
>>> > > > > --
>>> > > >
>>> > > > Jarek Potiuk
>>> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>>> > > >
>>> > > > M: +48 660 796 129 <+48660796129>
>>> > > > [image: Polidea] <https://www.polidea.com/>
>>> > > >
>>> > >
>>>
>>>

Re: Let's talk Airflow 2.0

Posted by Robin Edwards <ro...@bidnamic.com>.

> I have some thoughts on the Subdag (will open a new thread if necessary).
> Instead of having a separate child DAG, would it be better to chain all the
> tasks from the child dag to the parent dag and then drop the child dag?
> In this way, the whole child dag (actually just the tasks in it) will be
> respected by the scheduler the same as the parent dag.
>
> To have a similar zoom in/out in the UI, we can add a column to the
> task_instance table as a marker to group these tasks together when rendered
> in the UI.
>

+1

I think this is a brilliant idea and it would simplify things a lot.

We used to used subdags but ended up replacing them with a magic template
operator where you specified a start and end task to get round the subdag
issues.


Let me know how you guys think.

Thanks
> Bin
>
> On Thu, Apr 2, 2020 at 9:55 AM Daniel Imberman <da...@gmail.com>
> wrote:
>
> > Hello all,
> >
> > I've been reviewing This wiki page
> > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0> and I
> > wanted to discuss which of these features are still relevant/are blockers
> > for 2.0
> >
> > I figured we should start with the status of the high priority fixes
> >
> >
> >    1.
> > *Knative Executor (handled by KEDA) *The KEDA Autoscaler has accomplished
> >    everything that we would've hoped for from a KnativeExecutor. I think
> we
> >    can consider this work done.
> >    2. *Improve Webserver performance*
> >    @Kaxil Naik <ka...@gmail.com> It appears that DAG serialization
> is
> >    already merged? Are there other steps needed for webserver
> performance?
> >    3.
> > *Enhanced real-time UI *While this would make for a great "wow!"
> feature, I
> >    don't think this is necessary for a 2.0. Unless anyone wants to work
> on
> >    this in the short term :).
> >    4.
> > *Improve Scheduler performance and reliability @Ash Berlin-Taylor
> >    <as...@apache.org> *has been hard at work on scheduler HA, and we've
> >    already
> >    5.
> > *Extend/finish the API @Kamil Breguła <ka...@polidea.com> *I see
> >    that there is an active PR here
> >    <https://github.com/apache/airflow/pull/7549>, are we getting close
> on
> >    this?
> >    6.
> > *Production Docker image *The PR here should be merged hopefully today!
> >
> > Now there's a second section of "needs more info" tickets and I wanted to
> > see which of these we could triage for after 2.0
> >
> > Rework Subdags to be less "bolted-on" and more native to the scheduler
> > >
> > > There are all sorts of edge cases around subdags that result from the
> > > tasks in the subdag being run by another executor, instead of being
> > handled
> > > and scheduled by the core Scheduler. We should make the scheduler "see
> > in"
> > > to the Subdags and make it responsible for scheduling tasks. This
> should
> > > make subdags less error prone and more predictable. It may involve
> > > replacing/significantly changing the SubDagOperator
> >
> >
> > I think it's pretty damn crucial we fix subdags and backfills. I'm on the
> > fence about this one. On the one hand it could possibly wait. On the
> other
> > hand it would be embarrasing to release a 2.0 and still have this feature
> > broken
> >
> > Move (tested) components out of contrib folder
> > >
> > >
> >
> https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E
> > >
> >
> > I think this has been something we've actively worked on. Is this task
> > complete?
> >
> > Filter passwords/sensitive info from logs.
> > >
> > > Jenkins does this if the password comes from a connection - it would be
> > > good if we could do this too
> >
> >
> > While I agree that this is an important issue, I think if no one has
> picked
> > this up yet we should triage for a later release (I also don't think this
> > should break anything)
> >
> > Allow Backfill runs to be handled by the scheduler/triggered from UI
> > >
> > > It would be nice to not need console access to run airflow backfill,
> and
> > > to have not it not stop if the SSH session is closed.
> > >
> > > Lots of details to work out here though around how this would work,
> where
> > > would it show up in UI, priority of tasks, ways of reducing
> > > concurrency/load to allow normal tasks to run etc.
> >
> >
> > Once again I think this is a feature that's a bit embarrasing NOT to fix
> > for a 2.0 release, but also understand that there's only so much
> manpower.
> > I also imagine this would be a HUGE undertaking so I think this would
> push
> > back 2.0 significantly.
> >
> >
> > Rationalize HA in Connections
> > >
> > > Right now it is possible to create multiple connections with the same
> ID
> > > and *some*  Connections/hooks will support this and pick a random one
> > > from the list. This feature isn't well documented or understood (and
> the
> > > CLI doesn't support it as well as the UI for instance) so we should
> > examine
> > > if this makes sense, or if we should support it individually in certain
> > > connection types instead.
> >
> >
> > I honestly don't know enough about this to have a strong opinion on it
> >
> > Make setting up HTTPS connections easier/more expected
> > >
> > > It appears that @Kamil Breguła <ka...@polidea.com>  has an
> open
> > PR
> > <https://github.com/apache/airflow/pull/5239/files> for this one. @Kamil
> > Breguła <ka...@polidea.com> do you think this would be hard to
> > merge?
> >
> > Front end/"browser" testing
> > >
> > > The Airflow UI is non trivial and there have been a number of JS/html
> > bugs
> > > that could have been caught by better front-end testing.
> > >
> > > It has been suggested to look at Cypress for this over Selenium. What
> > ever
> > > we choose we need to pay careful attention to avoid slow or flakey UI
> > tests.
> > >
> >
> > This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've
> > been taking time to learn cypress recently, and once the airflow helm
> chart
> > is merged I think merging a set of integration/behavior/UI tests is
> crucial
> >
> > What does everyone think? Open to suggestions on this assessment!
> >
> >
> > On Tue, Mar 31, 2020 at 11:46 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Got it. Thanks Daniel for leading this
> > >
> > > On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman <
> > daniel.imberman@gmail.com>
> > > wrote:
> > >
> > >> I think including both is fine as long as the old one contains
> > >> deprecation warnings/force a feature flag to allow it
> > >> (e.g. —allow-deprecated)
> > >>
> > >> via Newton Mail
> > >> <
> >
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > >
> > >>
> > >> On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła <
> > >> kamil.bregula@polidea.com> wrote:
> > >>
> > >> On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com>
> wrote:
> > >>
> > >> > Also does the new API need to be feature complete or just enough
> > >> > functionality to warrant removing the existing experimental one.
> > >> >
> > >>
> > >> I think we should release at least one version that will contain the
> > >> new and old REST APIs simultaneously. It is not easy to upgrade two
> > >> complex systems at the same time. However, if we do this, some users
> > >> will have to do it. Older versions can be hidden behind the feature
> > >> gate. We can also add deprecation warnings.
> > >>
> > >> >
> > >> > R
> > >> >
> > >> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <
> > daniel.imberman@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Great! Hope to get a few more folx to give +1's but I think we
> have
> > a
> > >> good
> > >> > > path forward here :)
> > >> > >
> > >> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
> > >> Jarek.Potiuk@polidea.com>
> > >> > > wrote:
> > >> > >
> > >> > > > >
> > >> > > > >
> > >> > > > > I agree especially for larger-scale users migrations are a
> > >> difficult
> > >> > > > > process. Perhaps we can adopt something similar to a
> blockchain
> > >> fork
> > >> > > > (e.g.
> > >> > > > > determine X known airflow using companies, and start the
> > >> countdown as
> > >> > > > soon
> > >> > > > > as Y% of them migrate). I really just want to make sure we
> don't
> > >> end up
> > >> > > > > with a python2/3 situation. Even if we continue support it
> > should
> > >> only
> > >> > > be
> > >> > > > > for bugfixes and we should not add any new features into 1.10.
> > >> > > > >
> > >> > > >
> > >> > > > I think we are in perfect sync - I think feature-migration
> should
> > >> end
> > >> > > > almost immediately after we release 2.0. But bug-fixing should
> > >> continue
> > >> > > > for quite some time. On that front - having backport packages
> will
> > >> help
> > >> > > > with releasing "integrations" quite independently from 1.10/2.0
> > >> version
> > >> > > > (which I think is good for those who are - for this or another
> > >> reason -
> > >> > > > stuck on 2.0). On the other hand we should make sure that the
> > >> important
> > >> > > > stuff for 2.0 that is not "feature" is also backported to 1.10.
> > For
> > >> > > example
> > >> > > > a lot of recent performance improvements that we have now in 2.0
> > >> will
> > >> > > > be possible (and not that complex) to backport to 1.10. Some of
> > this
> > >> > > effort
> > >> > > > is actually easier to do in 2.0 and then apply to 1.10 in
> similar
> > >> fashion
> > >> > > > as it is easier to understand and reason about the 2.0 code now
> > when
> > >> > > > we have some refactoring/pylints etc in place. So we should make
> > >> sure
> > >> > > > we get the latest 1.10 to a "good" state - before we freeze it
> for
> > >> bugfix
> > >> > > > only.
> > >> > > > I know it might mean that some people will stay with 1.10 for
> > >> longer, but
> > >> > > > that's also OK for them. The reason to migrate to 2.0 should be
> > not
> > >> > > > performance but some important features (like API or HA) that
> come
> > >> > > > with it.
> > >> > > >
> > >> > > > I couldn't agree more :). If we can start people writing (close
> > to)
> > >> 2.0
> > >> > > > > compliant DAGs before the release of 2.0 that will make the
> > >> migration
> > >> > > > > process so much easier :).
> > >> > > > >
> > >> > > >
> > >> > > > Yeah. I even thought that we should write a
> > >> > > > "How good your DAGs are for 2.0" assessment tool.
> > >> > > >
> > >> > > >
> > >> > > > > If there aren't any extra steps or features that we need to
> add
> > >> (beyond
> > >> > > > the
> > >> > > > > ones discussed here), I think a good next step would be to
> > create
> > >> an
> > >> > > > > official checklist just so we can see all of these features in
> > one
> > >> > > place
> > >> > > > > (and hopefully start breaking them down into as small of
> changes
> > >> as
> > >> > > > > possible).
> > >> > > > >
> > >> > > > > Does that sound ok?
> > >> > > > >
> > >> > > >
> > >> > > > Perfectly OK for me!
> > >> > > >
> > >> > > >
> > >> > > > >
> > >> > > > > --
> > >> > > >
> > >> > > > Jarek Potiuk
> > >> > > > Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > >> > > >
> > >> > > > M: +48 660 796 129 <+48660796129>
> > >> > > > [image: Polidea] <https://www.polidea.com/>
> > >> > > >
> > >> > >
> > >>
> > >>
> >
>

Re: Let's talk Airflow 2.0

Posted by Xinbin Huang <bi...@gmail.com>.

>>Hi Bin,
>>
>> I had very similar thoughts about SubDags. End of day DAGs are just
graphs and graph joins are pretty easy operations. I’m gonna file a GitHub
issue and assign us both and let’s hash it out there :)

Sounds great! Looking forward to it.

On Thu, Apr 2, 2020 at 1:47 PM Daniel Imberman <da...@gmail.com>
wrote:

> Hi Bin,
>
> I had very similar thoughts about SubDags. End of day DAGs are just graphs
> and graph joins are pretty easy operations. I’m gonna file a GitHub issue
> and assign us both and let’s hash it out there :)
>
> via Newton Mail [
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> ]
> On Thu, Apr 2, 2020 at 1:32 PM, Xinbin Huang <bi...@gmail.com>
> wrote:
> >> I think it's pretty damn crucial we fix subdags and backfills. I'm on
> the
> >> fence about this one. On the one hand it could possibly wait. On the
> other
> >> hand it would be embarrasing to release a 2.0 and still have this
> feature
> >> broken
>
> I would really like to see subdags and backfills being fixed in 2.0 as I
> agree with Daniel that I would be quite embarrassing to see these features
> broken.
>
> I have some thoughts on the Subdag (will open a new thread if necessary).
> Instead of having a separate child DAG, would it be better to chain all the
> tasks from the child dag to the parent dag and then drop the child dag?
> In this way, the whole child dag (actually just the tasks in it) will be
> respected by the scheduler the same as the parent dag.
>
> To have a similar zoom in/out in the UI, we can add a column to the
> task_instance table as a marker to group these tasks together when rendered
> in the UI.
>
> Let me know how you guys think.
>
> Thanks
> Bin
>
> On Thu, Apr 2, 2020 at 9:55 AM Daniel Imberman <da...@gmail.com>
> wrote:
>
> > Hello all,
> >
> > I've been reviewing This wiki page
> > <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0> and I
> > wanted to discuss which of these features are still relevant/are blockers
> > for 2.0
> >
> > I figured we should start with the status of the high priority fixes
> >
> >
> > 1.
> > *Knative Executor (handled by KEDA) *The KEDA Autoscaler has accomplished
> > everything that we would've hoped for from a KnativeExecutor. I think we
> > can consider this work done.
> > 2. *Improve Webserver performance*
> > @Kaxil Naik <ka...@gmail.com> It appears that DAG serialization is
> > already merged? Are there other steps needed for webserver performance?
> > 3.
> > *Enhanced real-time UI *While this would make for a great "wow!"
> feature, I
> > don't think this is necessary for a 2.0. Unless anyone wants to work on
> > this in the short term :).
> > 4.
> > *Improve Scheduler performance and reliability @Ash Berlin-Taylor
> > <as...@apache.org> *has been hard at work on scheduler HA, and we've
> > already
> > 5.
> > *Extend/finish the API @Kamil Breguła <ka...@polidea.com> *I see
> > that there is an active PR here
> > <https://github.com/apache/airflow/pull/7549>, are we getting close on
> > this?
> > 6.
> > *Production Docker image *The PR here should be merged hopefully today!
> >
> > Now there's a second section of "needs more info" tickets and I wanted to
> > see which of these we could triage for after 2.0
> >
> > Rework Subdags to be less "bolted-on" and more native to the scheduler
> > >
> > > There are all sorts of edge cases around subdags that result from the
> > > tasks in the subdag being run by another executor, instead of being
> > handled
> > > and scheduled by the core Scheduler. We should make the scheduler "see
> > in"
> > > to the Subdags and make it responsible for scheduling tasks. This
> should
> > > make subdags less error prone and more predictable. It may involve
> > > replacing/significantly changing the SubDagOperator
> >
> >
> > I think it's pretty damn crucial we fix subdags and backfills. I'm on the
> > fence about this one. On the one hand it could possibly wait. On the
> other
> > hand it would be embarrasing to release a 2.0 and still have this feature
> > broken
> >
> > Move (tested) components out of contrib folder
> > >
> > >
> >
> https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E
> > >
> >
> > I think this has been something we've actively worked on. Is this task
> > complete?
> >
> > Filter passwords/sensitive info from logs.
> > >
> > > Jenkins does this if the password comes from a connection - it would be
> > > good if we could do this too
> >
> >
> > While I agree that this is an important issue, I think if no one has
> picked
> > this up yet we should triage for a later release (I also don't think this
> > should break anything)
> >
> > Allow Backfill runs to be handled by the scheduler/triggered from UI
> > >
> > > It would be nice to not need console access to run airflow backfill,
> and
> > > to have not it not stop if the SSH session is closed.
> > >
> > > Lots of details to work out here though around how this would work,
> where
> > > would it show up in UI, priority of tasks, ways of reducing
> > > concurrency/load to allow normal tasks to run etc.
> >
> >
> > Once again I think this is a feature that's a bit embarrasing NOT to fix
> > for a 2.0 release, but also understand that there's only so much
> manpower.
> > I also imagine this would be a HUGE undertaking so I think this would
> push
> > back 2.0 significantly.
> >
> >
> > Rationalize HA in Connections
> > >
> > > Right now it is possible to create multiple connections with the same
> ID
> > > and *some* Connections/hooks will support this and pick a random one
> > > from the list. This feature isn't well documented or understood (and
> the
> > > CLI doesn't support it as well as the UI for instance) so we should
> > examine
> > > if this makes sense, or if we should support it individually in certain
> > > connection types instead.
> >
> >
> > I honestly don't know enough about this to have a strong opinion on it
> >
> > Make setting up HTTPS connections easier/more expected
> > >
> > > It appears that @Kamil Breguła <ka...@polidea.com> has an open
> > PR
> > <https://github.com/apache/airflow/pull/5239/files> for this one. @Kamil
> > Breguła <ka...@polidea.com> do you think this would be hard to
> > merge?
> >
> > Front end/"browser" testing
> > >
> > > The Airflow UI is non trivial and there have been a number of JS/html
> > bugs
> > > that could have been caught by better front-end testing.
> > >
> > > It has been suggested to look at Cypress for this over Selenium. What
> > ever
> > > we choose we need to pay careful attention to avoid slow or flakey UI
> > tests.
> > >
> >
> > This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've
> > been taking time to learn cypress recently, and once the airflow helm
> chart
> > is merged I think merging a set of integration/behavior/UI tests is
> crucial
> >
> > What does everyone think? Open to suggestions on this assessment!
> >
> >
> > On Tue, Mar 31, 2020 at 11:46 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Got it. Thanks Daniel for leading this
> > >
> > > On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman <
> > daniel.imberman@gmail.com>
> > > wrote:
> > >
> > >> I think including both is fine as long as the old one contains
> > >> deprecation warnings/force a feature flag to allow it
> > >> (e.g. —allow-deprecated)
> > >>
> > >> via Newton Mail
> > >> <
> >
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > >
> > >>
> > >> On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła <
> > >> kamil.bregula@polidea.com> wrote:
> > >>
> > >> On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com>
> wrote:
> > >>
> > >> > Also does the new API need to be feature complete or just enough
> > >> > functionality to warrant removing the existing experimental one.
> > >> >
> > >>
> > >> I think we should release at least one version that will contain the
> > >> new and old REST APIs simultaneously. It is not easy to upgrade two
> > >> complex systems at the same time. However, if we do this, some users
> > >> will have to do it. Older versions can be hidden behind the feature
> > >> gate. We can also add deprecation warnings.
> > >>
> > >> >
> > >> > R
> > >> >
> > >> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <
> > daniel.imberman@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Great! Hope to get a few more folx to give +1's but I think we
> have
> > a
> > >> good
> > >> > > path forward here :)
> > >> > >
> > >> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
> > >> Jarek.Potiuk@polidea.com>
> > >> > > wrote:
> > >> > >
> > >> > > > >
> > >> > > > >
> > >> > > > > I agree especially for larger-scale users migrations are a
> > >> difficult
> > >> > > > > process. Perhaps we can adopt something similar to a
> blockchain
> > >> fork
> > >> > > > (e.g.
> > >> > > > > determine X known airflow using companies, and start the
> > >> countdown as
> > >> > > > soon
> > >> > > > > as Y% of them migrate). I really just want to make sure we
> don't
> > >> end up
> > >> > > > > with a python2/3 situation. Even if we continue support it
> > should
> > >> only
> > >> > > be
> > >> > > > > for bugfixes and we should not add any new features into 1.10.
> > >> > > > >
> > >> > > >
> > >> > > > I think we are in perfect sync - I think feature-migration
> should
> > >> end
> > >> > > > almost immediately after we release 2.0. But bug-fixing should
> > >> continue
> > >> > > > for quite some time. On that front - having backport packages
> will
> > >> help
> > >> > > > with releasing "integrations" quite independently from 1.10/2.0
> > >> version
> > >> > > > (which I think is good for those who are - for this or another
> > >> reason -
> > >> > > > stuck on 2.0). On the other hand we should make sure that the
> > >> important
> > >> > > > stuff for 2.0 that is not "feature" is also backported to 1.10.
> > For
> > >> > > example
> > >> > > > a lot of recent performance improvements that we have now in 2.0
> > >> will
> > >> > > > be possible (and not that complex) to backport to 1.10. Some of
> > this
> > >> > > effort
> > >> > > > is actually easier to do in 2.0 and then apply to 1.10 in
> similar
> > >> fashion
> > >> > > > as it is easier to understand and reason about the 2.0 code now
> > when
> > >> > > > we have some refactoring/pylints etc in place. So we should make
> > >> sure
> > >> > > > we get the latest 1.10 to a "good" state - before we freeze it
> for
> > >> bugfix
> > >> > > > only.
> > >> > > > I know it might mean that some people will stay with 1.10 for
> > >> longer, but
> > >> > > > that's also OK for them. The reason to migrate to 2.0 should be
> > not
> > >> > > > performance but some important features (like API or HA) that
> come
> > >> > > > with it.
> > >> > > >
> > >> > > > I couldn't agree more :). If we can start people writing (close
> > to)
> > >> 2.0
> > >> > > > > compliant DAGs before the release of 2.0 that will make the
> > >> migration
> > >> > > > > process so much easier :).
> > >> > > > >
> > >> > > >
> > >> > > > Yeah. I even thought that we should write a
> > >> > > > "How good your DAGs are for 2.0" assessment tool.
> > >> > > >
> > >> > > >
> > >> > > > > If there aren't any extra steps or features that we need to
> add
> > >> (beyond
> > >> > > > the
> > >> > > > > ones discussed here), I think a good next step would be to
> > create
> > >> an
> > >> > > > > official checklist just so we can see all of these features in
> > one
> > >> > > place
> > >> > > > > (and hopefully start breaking them down into as small of
> changes
> > >> as
> > >> > > > > possible).
> > >> > > > >
> > >> > > > > Does that sound ok?
> > >> > > > >
> > >> > > >
> > >> > > > Perfectly OK for me!
> > >> > > >
> > >> > > >
> > >> > > > >
> > >> > > > > --
> > >> > > >
> > >> > > > Jarek Potiuk
> > >> > > > Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > >> > > >
> > >> > > > M: +48 660 796 129 <+48660796129>
> > >> > > > [image: Polidea] <https://www.polidea.com/>
> > >> > > >
> > >> > >
> > >>
> > >>
> >

Re: Let's talk Airflow 2.0

Posted by Daniel Imberman <da...@gmail.com>.

Hi Bin,

I had very similar thoughts about SubDags. End of day DAGs are just graphs and graph joins are pretty easy operations. I’m gonna file a GitHub issue and assign us both and let’s hash it out there :)

via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2]
On Thu, Apr 2, 2020 at 1:32 PM, Xinbin Huang <bi...@gmail.com> wrote:
>> I think it's pretty damn crucial we fix subdags and backfills. I'm on
the
>> fence about this one. On the one hand it could possibly wait. On the
other
>> hand it would be embarrasing to release a 2.0 and still have this feature
>> broken

I would really like to see subdags and backfills being fixed in 2.0 as I
agree with Daniel that I would be quite embarrassing to see these features
broken.

I have some thoughts on the Subdag (will open a new thread if necessary).
Instead of having a separate child DAG, would it be better to chain all the
tasks from the child dag to the parent dag and then drop the child dag?
In this way, the whole child dag (actually just the tasks in it) will be
respected by the scheduler the same as the parent dag.

To have a similar zoom in/out in the UI, we can add a column to the
task_instance table as a marker to group these tasks together when rendered
in the UI.

Let me know how you guys think.

Thanks
Bin

On Thu, Apr 2, 2020 at 9:55 AM Daniel Imberman <da...@gmail.com>
wrote:

> Hello all,
>
> I've been reviewing This wiki page
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0> and I
> wanted to discuss which of these features are still relevant/are blockers
> for 2.0
>
> I figured we should start with the status of the high priority fixes
>
>
> 1.
> *Knative Executor (handled by KEDA) *The KEDA Autoscaler has accomplished
> everything that we would've hoped for from a KnativeExecutor. I think we
> can consider this work done.
> 2. *Improve Webserver performance*
> @Kaxil Naik <ka...@gmail.com> It appears that DAG serialization is
> already merged? Are there other steps needed for webserver performance?
> 3.
> *Enhanced real-time UI *While this would make for a great "wow!" feature, I
> don't think this is necessary for a 2.0. Unless anyone wants to work on
> this in the short term :).
> 4.
> *Improve Scheduler performance and reliability @Ash Berlin-Taylor
> <as...@apache.org> *has been hard at work on scheduler HA, and we've
> already
> 5.
> *Extend/finish the API @Kamil Breguła <ka...@polidea.com> *I see
> that there is an active PR here
> <https://github.com/apache/airflow/pull/7549>, are we getting close on
> this?
> 6.
> *Production Docker image *The PR here should be merged hopefully today!
>
> Now there's a second section of "needs more info" tickets and I wanted to
> see which of these we could triage for after 2.0
>
> Rework Subdags to be less "bolted-on" and more native to the scheduler
> >
> > There are all sorts of edge cases around subdags that result from the
> > tasks in the subdag being run by another executor, instead of being
> handled
> > and scheduled by the core Scheduler. We should make the scheduler "see
> in"
> > to the Subdags and make it responsible for scheduling tasks. This should
> > make subdags less error prone and more predictable. It may involve
> > replacing/significantly changing the SubDagOperator
>
>
> I think it's pretty damn crucial we fix subdags and backfills. I'm on the
> fence about this one. On the one hand it could possibly wait. On the other
> hand it would be embarrasing to release a 2.0 and still have this feature
> broken
>
> Move (tested) components out of contrib folder
> >
> >
> https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E
> >
>
> I think this has been something we've actively worked on. Is this task
> complete?
>
> Filter passwords/sensitive info from logs.
> >
> > Jenkins does this if the password comes from a connection - it would be
> > good if we could do this too
>
>
> While I agree that this is an important issue, I think if no one has picked
> this up yet we should triage for a later release (I also don't think this
> should break anything)
>
> Allow Backfill runs to be handled by the scheduler/triggered from UI
> >
> > It would be nice to not need console access to run airflow backfill, and
> > to have not it not stop if the SSH session is closed.
> >
> > Lots of details to work out here though around how this would work, where
> > would it show up in UI, priority of tasks, ways of reducing
> > concurrency/load to allow normal tasks to run etc.
>
>
> Once again I think this is a feature that's a bit embarrasing NOT to fix
> for a 2.0 release, but also understand that there's only so much manpower.
> I also imagine this would be a HUGE undertaking so I think this would push
> back 2.0 significantly.
>
>
> Rationalize HA in Connections
> >
> > Right now it is possible to create multiple connections with the same ID
> > and *some* Connections/hooks will support this and pick a random one
> > from the list. This feature isn't well documented or understood (and the
> > CLI doesn't support it as well as the UI for instance) so we should
> examine
> > if this makes sense, or if we should support it individually in certain
> > connection types instead.
>
>
> I honestly don't know enough about this to have a strong opinion on it
>
> Make setting up HTTPS connections easier/more expected
> >
> > It appears that @Kamil Breguła <ka...@polidea.com> has an open
> PR
> <https://github.com/apache/airflow/pull/5239/files> for this one. @Kamil
> Breguła <ka...@polidea.com> do you think this would be hard to
> merge?
>
> Front end/"browser" testing
> >
> > The Airflow UI is non trivial and there have been a number of JS/html
> bugs
> > that could have been caught by better front-end testing.
> >
> > It has been suggested to look at Cypress for this over Selenium. What
> ever
> > we choose we need to pay careful attention to avoid slow or flakey UI
> tests.
> >
>
> This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've
> been taking time to learn cypress recently, and once the airflow helm chart
> is merged I think merging a set of integration/behavior/UI tests is crucial
>
> What does everyone think? Open to suggestions on this assessment!
>
>
> On Tue, Mar 31, 2020 at 11:46 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Got it. Thanks Daniel for leading this
> >
> > On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman <
> daniel.imberman@gmail.com>
> > wrote:
> >
> >> I think including both is fine as long as the old one contains
> >> deprecation warnings/force a feature flag to allow it
> >> (e.g. —allow-deprecated)
> >>
> >> via Newton Mail
> >> <
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> >
> >>
> >> On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła <
> >> kamil.bregula@polidea.com> wrote:
> >>
> >> On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com> wrote:
> >>
> >> > Also does the new API need to be feature complete or just enough
> >> > functionality to warrant removing the existing experimental one.
> >> >
> >>
> >> I think we should release at least one version that will contain the
> >> new and old REST APIs simultaneously. It is not easy to upgrade two
> >> complex systems at the same time. However, if we do this, some users
> >> will have to do it. Older versions can be hidden behind the feature
> >> gate. We can also add deprecation warnings.
> >>
> >> >
> >> > R
> >> >
> >> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <
> daniel.imberman@gmail.com>
> >> > wrote:
> >> >
> >> > > Great! Hope to get a few more folx to give +1's but I think we have
> a
> >> good
> >> > > path forward here :)
> >> > >
> >> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
> >> Jarek.Potiuk@polidea.com>
> >> > > wrote:
> >> > >
> >> > > > >
> >> > > > >
> >> > > > > I agree especially for larger-scale users migrations are a
> >> difficult
> >> > > > > process. Perhaps we can adopt something similar to a blockchain
> >> fork
> >> > > > (e.g.
> >> > > > > determine X known airflow using companies, and start the
> >> countdown as
> >> > > > soon
> >> > > > > as Y% of them migrate). I really just want to make sure we don't
> >> end up
> >> > > > > with a python2/3 situation. Even if we continue support it
> should
> >> only
> >> > > be
> >> > > > > for bugfixes and we should not add any new features into 1.10.
> >> > > > >
> >> > > >
> >> > > > I think we are in perfect sync - I think feature-migration should
> >> end
> >> > > > almost immediately after we release 2.0. But bug-fixing should
> >> continue
> >> > > > for quite some time. On that front - having backport packages will
> >> help
> >> > > > with releasing "integrations" quite independently from 1.10/2.0
> >> version
> >> > > > (which I think is good for those who are - for this or another
> >> reason -
> >> > > > stuck on 2.0). On the other hand we should make sure that the
> >> important
> >> > > > stuff for 2.0 that is not "feature" is also backported to 1.10.
> For
> >> > > example
> >> > > > a lot of recent performance improvements that we have now in 2.0
> >> will
> >> > > > be possible (and not that complex) to backport to 1.10. Some of
> this
> >> > > effort
> >> > > > is actually easier to do in 2.0 and then apply to 1.10 in similar
> >> fashion
> >> > > > as it is easier to understand and reason about the 2.0 code now
> when
> >> > > > we have some refactoring/pylints etc in place. So we should make
> >> sure
> >> > > > we get the latest 1.10 to a "good" state - before we freeze it for
> >> bugfix
> >> > > > only.
> >> > > > I know it might mean that some people will stay with 1.10 for
> >> longer, but
> >> > > > that's also OK for them. The reason to migrate to 2.0 should be
> not
> >> > > > performance but some important features (like API or HA) that come
> >> > > > with it.
> >> > > >
> >> > > > I couldn't agree more :). If we can start people writing (close
> to)
> >> 2.0
> >> > > > > compliant DAGs before the release of 2.0 that will make the
> >> migration
> >> > > > > process so much easier :).
> >> > > > >
> >> > > >
> >> > > > Yeah. I even thought that we should write a
> >> > > > "How good your DAGs are for 2.0" assessment tool.
> >> > > >
> >> > > >
> >> > > > > If there aren't any extra steps or features that we need to add
> >> (beyond
> >> > > > the
> >> > > > > ones discussed here), I think a good next step would be to
> create
> >> an
> >> > > > > official checklist just so we can see all of these features in
> one
> >> > > place
> >> > > > > (and hopefully start breaking them down into as small of changes
> >> as
> >> > > > > possible).
> >> > > > >
> >> > > > > Does that sound ok?
> >> > > > >
> >> > > >
> >> > > > Perfectly OK for me!
> >> > > >
> >> > > >
> >> > > > >
> >> > > > > --
> >> > > >
> >> > > > Jarek Potiuk
> >> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >> > > >
> >> > > > M: +48 660 796 129 <+48660796129>
> >> > > > [image: Polidea] <https://www.polidea.com/>
> >> > > >
> >> > >
> >>
> >>
>

Re: Let's talk Airflow 2.0

Posted by Xinbin Huang <bi...@gmail.com>.

>>  I think it's pretty damn crucial we fix subdags and backfills. I'm on
the
>> fence about this one. On the one hand it could possibly wait. On the
other
>> hand it would be embarrasing to release a 2.0 and still have this feature
>> broken

I would really like to see subdags and backfills being fixed in 2.0 as I
agree with Daniel that I would be quite embarrassing to see these features
broken.

I have some thoughts on the Subdag (will open a new thread if necessary).
Instead of having a separate child DAG, would it be better to chain all the
tasks from the child dag to the parent dag and then drop the child dag?
In this way, the whole child dag (actually just the tasks in it) will be
respected by the scheduler the same as the parent dag.

To have a similar zoom in/out in the UI, we can add a column to the
task_instance table as a marker to group these tasks together when rendered
in the UI.

Let me know how you guys think.

Thanks
Bin

On Thu, Apr 2, 2020 at 9:55 AM Daniel Imberman <da...@gmail.com>
wrote:

> Hello all,
>
> I've been reviewing This wiki page
> <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0> and I
> wanted to discuss which of these features are still relevant/are blockers
> for 2.0
>
> I figured we should start with the status of the high priority fixes
>
>
>    1.
> *Knative Executor (handled by KEDA) *The KEDA Autoscaler has accomplished
>    everything that we would've hoped for from a KnativeExecutor. I think we
>    can consider this work done.
>    2. *Improve Webserver performance*
>    @Kaxil Naik <ka...@gmail.com> It appears that DAG serialization is
>    already merged? Are there other steps needed for webserver performance?
>    3.
> *Enhanced real-time UI *While this would make for a great "wow!" feature, I
>    don't think this is necessary for a 2.0. Unless anyone wants to work on
>    this in the short term :).
>    4.
> *Improve Scheduler performance and reliability @Ash Berlin-Taylor
>    <as...@apache.org> *has been hard at work on scheduler HA, and we've
>    already
>    5.
> *Extend/finish the API @Kamil Breguła <ka...@polidea.com> *I see
>    that there is an active PR here
>    <https://github.com/apache/airflow/pull/7549>, are we getting close on
>    this?
>    6.
> *Production Docker image *The PR here should be merged hopefully today!
>
> Now there's a second section of "needs more info" tickets and I wanted to
> see which of these we could triage for after 2.0
>
> Rework Subdags to be less "bolted-on" and more native to the scheduler
> >
> > There are all sorts of edge cases around subdags that result from the
> > tasks in the subdag being run by another executor, instead of being
> handled
> > and scheduled by the core Scheduler. We should make the scheduler "see
> in"
> > to the Subdags and make it responsible for scheduling tasks. This should
> > make subdags less error prone and more predictable. It may involve
> > replacing/significantly changing the SubDagOperator
>
>
> I think it's pretty damn crucial we fix subdags and backfills. I'm on the
> fence about this one. On the one hand it could possibly wait. On the other
> hand it would be embarrasing to release a 2.0 and still have this feature
> broken
>
> Move (tested) components out of contrib folder
> >
> >
> https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E
> >
>
> I think this has been something we've actively worked on. Is this task
> complete?
>
> Filter passwords/sensitive info from logs.
> >
> > Jenkins does this if the password comes from a connection - it would be
> > good if we could do this too
>
>
> While I agree that this is an important issue, I think if no one has picked
> this up yet we should triage for a later release (I also don't think this
> should break anything)
>
> Allow Backfill runs to be handled by the scheduler/triggered from UI
> >
> > It would be nice to not need console access to run airflow backfill, and
> > to have not it not stop if the SSH session is closed.
> >
> > Lots of details to work out here though around how this would work, where
> > would it show up in UI, priority of tasks, ways of reducing
> > concurrency/load to allow normal tasks to run etc.
>
>
> Once again I think this is a feature that's a bit embarrasing NOT to fix
> for a 2.0 release, but also understand that there's only so much manpower.
> I also imagine this would be a HUGE undertaking so I think this would push
> back 2.0 significantly.
>
>
> Rationalize HA in Connections
> >
> > Right now it is possible to create multiple connections with the same ID
> > and *some*  Connections/hooks will support this and pick a random one
> > from the list. This feature isn't well documented or understood (and the
> > CLI doesn't support it as well as the UI for instance) so we should
> examine
> > if this makes sense, or if we should support it individually in certain
> > connection types instead.
>
>
> I honestly don't know enough about this to have a strong opinion on it
>
> Make setting up HTTPS connections easier/more expected
> >
> > It appears that @Kamil Breguła <ka...@polidea.com>  has an open
> PR
> <https://github.com/apache/airflow/pull/5239/files> for this one. @Kamil
> Breguła <ka...@polidea.com> do you think this would be hard to
> merge?
>
> Front end/"browser" testing
> >
> > The Airflow UI is non trivial and there have been a number of JS/html
> bugs
> > that could have been caught by better front-end testing.
> >
> > It has been suggested to look at Cypress for this over Selenium. What
> ever
> > we choose we need to pay careful attention to avoid slow or flakey UI
> tests.
> >
>
> This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've
> been taking time to learn cypress recently, and once the airflow helm chart
> is merged I think merging a set of integration/behavior/UI tests is crucial
>
> What does everyone think? Open to suggestions on this assessment!
>
>
> On Tue, Mar 31, 2020 at 11:46 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Got it. Thanks Daniel for leading this
> >
> > On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman <
> daniel.imberman@gmail.com>
> > wrote:
> >
> >> I think including both is fine as long as the old one contains
> >> deprecation warnings/force a feature flag to allow it
> >> (e.g. —allow-deprecated)
> >>
> >> via Newton Mail
> >> <
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> >
> >>
> >> On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła <
> >> kamil.bregula@polidea.com> wrote:
> >>
> >> On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com> wrote:
> >>
> >> > Also does the new API need to be feature complete or just enough
> >> > functionality to warrant removing the existing experimental one.
> >> >
> >>
> >> I think we should release at least one version that will contain the
> >> new and old REST APIs simultaneously. It is not easy to upgrade two
> >> complex systems at the same time. However, if we do this, some users
> >> will have to do it. Older versions can be hidden behind the feature
> >> gate. We can also add deprecation warnings.
> >>
> >> >
> >> > R
> >> >
> >> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <
> daniel.imberman@gmail.com>
> >> > wrote:
> >> >
> >> > > Great! Hope to get a few more folx to give +1's but I think we have
> a
> >> good
> >> > > path forward here :)
> >> > >
> >> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
> >> Jarek.Potiuk@polidea.com>
> >> > > wrote:
> >> > >
> >> > > > >
> >> > > > >
> >> > > > > I agree especially for larger-scale users migrations are a
> >> difficult
> >> > > > > process. Perhaps we can adopt something similar to a blockchain
> >> fork
> >> > > > (e.g.
> >> > > > > determine X known airflow using companies, and start the
> >> countdown as
> >> > > > soon
> >> > > > > as Y% of them migrate). I really just want to make sure we don't
> >> end up
> >> > > > > with a python2/3 situation. Even if we continue support it
> should
> >> only
> >> > > be
> >> > > > > for bugfixes and we should not add any new features into 1.10.
> >> > > > >
> >> > > >
> >> > > > I think we are in perfect sync - I think feature-migration should
> >> end
> >> > > > almost immediately after we release 2.0. But bug-fixing should
> >> continue
> >> > > > for quite some time. On that front - having backport packages will
> >> help
> >> > > > with releasing "integrations" quite independently from 1.10/2.0
> >> version
> >> > > > (which I think is good for those who are - for this or another
> >> reason -
> >> > > > stuck on 2.0). On the other hand we should make sure that the
> >> important
> >> > > > stuff for 2.0 that is not "feature" is also backported to 1.10.
> For
> >> > > example
> >> > > > a lot of recent performance improvements that we have now in 2.0
> >> will
> >> > > > be possible (and not that complex) to backport to 1.10. Some of
> this
> >> > > effort
> >> > > > is actually easier to do in 2.0 and then apply to 1.10 in similar
> >> fashion
> >> > > > as it is easier to understand and reason about the 2.0 code now
> when
> >> > > > we have some refactoring/pylints etc in place. So we should make
> >> sure
> >> > > > we get the latest 1.10 to a "good" state - before we freeze it for
> >> bugfix
> >> > > > only.
> >> > > > I know it might mean that some people will stay with 1.10 for
> >> longer, but
> >> > > > that's also OK for them. The reason to migrate to 2.0 should be
> not
> >> > > > performance but some important features (like API or HA) that come
> >> > > > with it.
> >> > > >
> >> > > > I couldn't agree more :). If we can start people writing (close
> to)
> >> 2.0
> >> > > > > compliant DAGs before the release of 2.0 that will make the
> >> migration
> >> > > > > process so much easier :).
> >> > > > >
> >> > > >
> >> > > > Yeah. I even thought that we should write a
> >> > > > "How good your DAGs are for 2.0" assessment tool.
> >> > > >
> >> > > >
> >> > > > > If there aren't any extra steps or features that we need to add
> >> (beyond
> >> > > > the
> >> > > > > ones discussed here), I think a good next step would be to
> create
> >> an
> >> > > > > official checklist just so we can see all of these features in
> one
> >> > > place
> >> > > > > (and hopefully start breaking them down into as small of changes
> >> as
> >> > > > > possible).
> >> > > > >
> >> > > > > Does that sound ok?
> >> > > > >
> >> > > >
> >> > > > Perfectly OK for me!
> >> > > >
> >> > > >
> >> > > > >
> >> > > > > --
> >> > > >
> >> > > > Jarek Potiuk
> >> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >> > > >
> >> > > > M: +48 660 796 129 <+48660796129>
> >> > > > [image: Polidea] <https://www.polidea.com/>
> >> > > >
> >> > >
> >>
> >>
>

Re: Let's talk Airflow 2.0

Posted by Daniel Imberman <da...@gmail.com>.

Hello all,

I've been reviewing This wiki page
<https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0> and I
wanted to discuss which of these features are still relevant/are blockers
for 2.0

I figured we should start with the status of the high priority fixes


   1.
*Knative Executor (handled by KEDA) *The KEDA Autoscaler has accomplished
   everything that we would've hoped for from a KnativeExecutor. I think we
   can consider this work done.
   2. *Improve Webserver performance*
   @Kaxil Naik <ka...@gmail.com> It appears that DAG serialization is
   already merged? Are there other steps needed for webserver performance?
   3.
*Enhanced real-time UI *While this would make for a great "wow!" feature, I
   don't think this is necessary for a 2.0. Unless anyone wants to work on
   this in the short term :).
   4.
*Improve Scheduler performance and reliability @Ash Berlin-Taylor
   <as...@apache.org> *has been hard at work on scheduler HA, and we've
   already
   5.
*Extend/finish the API @Kamil Breguła <ka...@polidea.com> *I see
   that there is an active PR here
   <https://github.com/apache/airflow/pull/7549>, are we getting close on
   this?
   6.
*Production Docker image *The PR here should be merged hopefully today!

Now there's a second section of "needs more info" tickets and I wanted to
see which of these we could triage for after 2.0

Rework Subdags to be less "bolted-on" and more native to the scheduler
>
> There are all sorts of edge cases around subdags that result from the
> tasks in the subdag being run by another executor, instead of being handled
> and scheduled by the core Scheduler. We should make the scheduler "see in"
> to the Subdags and make it responsible for scheduling tasks. This should
> make subdags less error prone and more predictable. It may involve
> replacing/significantly changing the SubDagOperator


I think it's pretty damn crucial we fix subdags and backfills. I'm on the
fence about this one. On the one hand it could possibly wait. On the other
hand it would be embarrasing to release a 2.0 and still have this feature
broken

Move (tested) components out of contrib folder
>
> https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E
>

I think this has been something we've actively worked on. Is this task
complete?

Filter passwords/sensitive info from logs.
>
> Jenkins does this if the password comes from a connection - it would be
> good if we could do this too


While I agree that this is an important issue, I think if no one has picked
this up yet we should triage for a later release (I also don't think this
should break anything)

Allow Backfill runs to be handled by the scheduler/triggered from UI
>
> It would be nice to not need console access to run airflow backfill, and
> to have not it not stop if the SSH session is closed.
>
> Lots of details to work out here though around how this would work, where
> would it show up in UI, priority of tasks, ways of reducing
> concurrency/load to allow normal tasks to run etc.


Once again I think this is a feature that's a bit embarrasing NOT to fix
for a 2.0 release, but also understand that there's only so much manpower.
I also imagine this would be a HUGE undertaking so I think this would push
back 2.0 significantly.


Rationalize HA in Connections
>
> Right now it is possible to create multiple connections with the same ID
> and *some*  Connections/hooks will support this and pick a random one
> from the list. This feature isn't well documented or understood (and the
> CLI doesn't support it as well as the UI for instance) so we should examine
> if this makes sense, or if we should support it individually in certain
> connection types instead.


I honestly don't know enough about this to have a strong opinion on it

Make setting up HTTPS connections easier/more expected
>
> It appears that @Kamil Breguła <ka...@polidea.com>  has an open PR
<https://github.com/apache/airflow/pull/5239/files> for this one. @Kamil
Breguła <ka...@polidea.com> do you think this would be hard to
merge?

Front end/"browser" testing
>
> The Airflow UI is non trivial and there have been a number of JS/html bugs
> that could have been caught by better front-end testing.
>
> It has been suggested to look at Cypress for this over Selenium. What ever
> we choose we need to pay careful attention to avoid slow or flakey UI tests.
>

This, to me, is a crucial step in ensuring a smooth 2.0 transition. I've
been taking time to learn cypress recently, and once the airflow helm chart
is merged I think merging a set of integration/behavior/UI tests is crucial

What does everyone think? Open to suggestions on this assessment!


On Tue, Mar 31, 2020 at 11:46 AM Kaxil Naik <ka...@gmail.com> wrote:

> Got it. Thanks Daniel for leading this
>
> On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> I think including both is fine as long as the old one contains
>> deprecation warnings/force a feature flag to allow it
>> (e.g. —allow-deprecated)
>>
>> via Newton Mail
>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2>
>>
>> On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła <
>> kamil.bregula@polidea.com> wrote:
>>
>> On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com> wrote:
>>
>> > Also does the new API need to be feature complete or just enough
>> > functionality to warrant removing the existing experimental one.
>> >
>>
>> I think we should release at least one version that will contain the
>> new and old REST APIs simultaneously. It is not easy to upgrade two
>> complex systems at the same time. However, if we do this, some users
>> will have to do it. Older versions can be hidden behind the feature
>> gate. We can also add deprecation warnings.
>>
>> >
>> > R
>> >
>> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <da...@gmail.com>
>> > wrote:
>> >
>> > > Great! Hope to get a few more folx to give +1's but I think we have a
>> good
>> > > path forward here :)
>> > >
>> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>
>> > > wrote:
>> > >
>> > > > >
>> > > > >
>> > > > > I agree especially for larger-scale users migrations are a
>> difficult
>> > > > > process. Perhaps we can adopt something similar to a blockchain
>> fork
>> > > > (e.g.
>> > > > > determine X known airflow using companies, and start the
>> countdown as
>> > > > soon
>> > > > > as Y% of them migrate). I really just want to make sure we don't
>> end up
>> > > > > with a python2/3 situation. Even if we continue support it should
>> only
>> > > be
>> > > > > for bugfixes and we should not add any new features into 1.10.
>> > > > >
>> > > >
>> > > > I think we are in perfect sync - I think feature-migration should
>> end
>> > > > almost immediately after we release 2.0. But bug-fixing should
>> continue
>> > > > for quite some time. On that front - having backport packages will
>> help
>> > > > with releasing "integrations" quite independently from 1.10/2.0
>> version
>> > > > (which I think is good for those who are - for this or another
>> reason -
>> > > > stuck on 2.0). On the other hand we should make sure that the
>> important
>> > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For
>> > > example
>> > > > a lot of recent performance improvements that we have now in 2.0
>> will
>> > > > be possible (and not that complex) to backport to 1.10. Some of this
>> > > effort
>> > > > is actually easier to do in 2.0 and then apply to 1.10 in similar
>> fashion
>> > > > as it is easier to understand and reason about the 2.0 code now when
>> > > > we have some refactoring/pylints etc in place. So we should make
>> sure
>> > > > we get the latest 1.10 to a "good" state - before we freeze it for
>> bugfix
>> > > > only.
>> > > > I know it might mean that some people will stay with 1.10 for
>> longer, but
>> > > > that's also OK for them. The reason to migrate to 2.0 should be not
>> > > > performance but some important features (like API or HA) that come
>> > > > with it.
>> > > >
>> > > > I couldn't agree more :). If we can start people writing (close to)
>> 2.0
>> > > > > compliant DAGs before the release of 2.0 that will make the
>> migration
>> > > > > process so much easier :).
>> > > > >
>> > > >
>> > > > Yeah. I even thought that we should write a
>> > > > "How good your DAGs are for 2.0" assessment tool.
>> > > >
>> > > >
>> > > > > If there aren't any extra steps or features that we need to add
>> (beyond
>> > > > the
>> > > > > ones discussed here), I think a good next step would be to create
>> an
>> > > > > official checklist just so we can see all of these features in one
>> > > place
>> > > > > (and hopefully start breaking them down into as small of changes
>> as
>> > > > > possible).
>> > > > >
>> > > > > Does that sound ok?
>> > > > >
>> > > >
>> > > > Perfectly OK for me!
>> > > >
>> > > >
>> > > > >
>> > > > > --
>> > > >
>> > > > Jarek Potiuk
>> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > > >
>> > > > M: +48 660 796 129 <+48660796129>
>> > > > [image: Polidea] <https://www.polidea.com/>
>> > > >
>> > >
>>
>>

Re: Let's talk Airflow 2.0

Posted by Kaxil Naik <ka...@gmail.com>.

Got it. Thanks Daniel for leading this

On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman <da...@gmail.com>
wrote:

> I think including both is fine as long as the old one contains deprecation
> warnings/force a feature flag to allow it (e.g. —allow-deprecated)
>
> via Newton Mail
> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2>
>
> On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła <ka...@polidea.com>
> wrote:
>
> On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com> wrote:
>
> > Also does the new API need to be feature complete or just enough
> > functionality to warrant removing the existing experimental one.
> >
>
> I think we should release at least one version that will contain the
> new and old REST APIs simultaneously. It is not easy to upgrade two
> complex systems at the same time. However, if we do this, some users
> will have to do it. Older versions can be hidden behind the feature
> gate. We can also add deprecation warnings.
>
> >
> > R
> >
> > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <da...@gmail.com>
> > wrote:
> >
> > > Great! Hope to get a few more folx to give +1's but I think we have a
> good
> > > path forward here :)
> > >
> > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > wrote:
> > >
> > > > >
> > > > >
> > > > > I agree especially for larger-scale users migrations are a
> difficult
> > > > > process. Perhaps we can adopt something similar to a blockchain
> fork
> > > > (e.g.
> > > > > determine X known airflow using companies, and start the countdown
> as
> > > > soon
> > > > > as Y% of them migrate). I really just want to make sure we don't
> end up
> > > > > with a python2/3 situation. Even if we continue support it should
> only
> > > be
> > > > > for bugfixes and we should not add any new features into 1.10.
> > > > >
> > > >
> > > > I think we are in perfect sync - I think feature-migration should end
> > > > almost immediately after we release 2.0. But bug-fixing should
> continue
> > > > for quite some time. On that front - having backport packages will
> help
> > > > with releasing "integrations" quite independently from 1.10/2.0
> version
> > > > (which I think is good for those who are - for this or another
> reason -
> > > > stuck on 2.0). On the other hand we should make sure that the
> important
> > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For
> > > example
> > > > a lot of recent performance improvements that we have now in 2.0 will
> > > > be possible (and not that complex) to backport to 1.10. Some of this
> > > effort
> > > > is actually easier to do in 2.0 and then apply to 1.10 in similar
> fashion
> > > > as it is easier to understand and reason about the 2.0 code now when
> > > > we have some refactoring/pylints etc in place. So we should make sure
> > > > we get the latest 1.10 to a "good" state - before we freeze it for
> bugfix
> > > > only.
> > > > I know it might mean that some people will stay with 1.10 for
> longer, but
> > > > that's also OK for them. The reason to migrate to 2.0 should be not
> > > > performance but some important features (like API or HA) that come
> > > > with it.
> > > >
> > > > I couldn't agree more :). If we can start people writing (close to)
> 2.0
> > > > > compliant DAGs before the release of 2.0 that will make the
> migration
> > > > > process so much easier :).
> > > > >
> > > >
> > > > Yeah. I even thought that we should write a
> > > > "How good your DAGs are for 2.0" assessment tool.
> > > >
> > > >
> > > > > If there aren't any extra steps or features that we need to add
> (beyond
> > > > the
> > > > > ones discussed here), I think a good next step would be to create
> an
> > > > > official checklist just so we can see all of these features in one
> > > place
> > > > > (and hopefully start breaking them down into as small of changes as
> > > > > possible).
> > > > >
> > > > > Does that sound ok?
> > > > >
> > > >
> > > > Perfectly OK for me!
> > > >
> > > >
> > > > >
> > > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > >
>
>

Re: Let's talk Airflow 2.0

Posted by Daniel Imberman <da...@gmail.com>.

I think including both is fine as long as the old one contains deprecation warnings/force a feature flag to allow it (e.g. —allow-deprecated)
via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2]
On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła <ka...@polidea.com> wrote:
On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com> wrote:

> Also does the new API need to be feature complete or just enough
> functionality to warrant removing the existing experimental one.
>

I think we should release at least one version that will contain the
new and old REST APIs simultaneously. It is not easy to upgrade two
complex systems at the same time. However, if we do this, some users
will have to do it. Older versions can be hidden behind the feature
gate. We can also add deprecation warnings.

>
> R
>
> On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <da...@gmail.com>
> wrote:
>
> > Great! Hope to get a few more folx to give +1's but I think we have a good
> > path forward here :)
> >
> > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > >
> > > >
> > > > I agree especially for larger-scale users migrations are a difficult
> > > > process. Perhaps we can adopt something similar to a blockchain fork
> > > (e.g.
> > > > determine X known airflow using companies, and start the countdown as
> > > soon
> > > > as Y% of them migrate). I really just want to make sure we don't end up
> > > > with a python2/3 situation. Even if we continue support it should only
> > be
> > > > for bugfixes and we should not add any new features into 1.10.
> > > >
> > >
> > > I think we are in perfect sync - I think feature-migration should end
> > > almost immediately after we release 2.0. But bug-fixing should continue
> > > for quite some time. On that front - having backport packages will help
> > > with releasing "integrations" quite independently from 1.10/2.0 version
> > > (which I think is good for those who are - for this or another reason -
> > > stuck on 2.0). On the other hand we should make sure that the important
> > > stuff for 2.0 that is not "feature" is also backported to 1.10. For
> > example
> > > a lot of recent performance improvements that we have now in 2.0 will
> > > be possible (and not that complex) to backport to 1.10. Some of this
> > effort
> > > is actually easier to do in 2.0 and then apply to 1.10 in similar fashion
> > > as it is easier to understand and reason about the 2.0 code now when
> > > we have some refactoring/pylints etc in place. So we should make sure
> > > we get the latest 1.10 to a "good" state - before we freeze it for bugfix
> > > only.
> > > I know it might mean that some people will stay with 1.10 for longer, but
> > > that's also OK for them. The reason to migrate to 2.0 should be not
> > > performance but some important features (like API or HA) that come
> > > with it.
> > >
> > > I couldn't agree more :). If we can start people writing (close to) 2.0
> > > > compliant DAGs before the release of 2.0 that will make the migration
> > > > process so much easier :).
> > > >
> > >
> > > Yeah. I even thought that we should write a
> > > "How good your DAGs are for 2.0" assessment tool.
> > >
> > >
> > > > If there aren't any extra steps or features that we need to add (beyond
> > > the
> > > > ones discussed here), I think a good next step would be to create an
> > > > official checklist just so we can see all of these features in one
> > place
> > > > (and hopefully start breaking them down into as small of changes as
> > > > possible).
> > > >
> > > > Does that sound ok?
> > > >
> > >
> > > Perfectly OK for me!
> > >
> > >
> > > >
> > > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> >

Re: Let's talk Airflow 2.0

Posted by Kamil Breguła <ka...@polidea.com>.

On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards <ro...@bidnamic.com> wrote:

> Also does the new API need to be feature complete or just enough
> functionality to warrant removing the existing experimental one.
>

I think we should release at least one version that will contain the
new and old REST APIs simultaneously. It is not easy to upgrade two
complex systems at the same time. However, if we do this, some users
will have to do it. Older versions can be hidden behind the feature
gate. We can also add deprecation warnings.

>
> R
>
> On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <da...@gmail.com>
> wrote:
>
> > Great! Hope to get a few more folx to give +1's but I think we have a good
> > path forward here :)
> >
> > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > >
> > > >
> > > > I agree especially for larger-scale users migrations are a difficult
> > > > process. Perhaps we can adopt something similar to a blockchain fork
> > > (e.g.
> > > > determine X known airflow using companies, and start the countdown as
> > > soon
> > > > as Y% of them migrate). I really just want to make sure we don't end up
> > > > with a python2/3 situation. Even if we continue support it should only
> > be
> > > > for bugfixes and we should not add any new features into 1.10.
> > > >
> > >
> > > I think we are in perfect sync  - I think feature-migration should end
> > > almost immediately after we release 2.0. But bug-fixing should continue
> > > for quite some time. On that front - having backport packages will help
> > > with releasing "integrations" quite independently from 1.10/2.0 version
> > > (which I think is good for those who are - for this or another reason -
> > > stuck on 2.0). On the other hand we should make sure that the important
> > > stuff for 2.0 that is not "feature" is also backported to 1.10. For
> > example
> > > a lot of recent performance improvements that we have now in 2.0 will
> > > be possible (and not that complex) to backport to 1.10. Some of this
> > effort
> > > is actually easier to do in 2.0 and then apply to 1.10 in similar fashion
> > > as it is easier to understand and reason about the 2.0 code now when
> > > we have some refactoring/pylints etc in place. So we should make sure
> > > we get the latest 1.10 to a "good" state - before we freeze it for bugfix
> > > only.
> > > I know it might mean that some people will stay with 1.10 for longer, but
> > > that's also OK for them. The reason to migrate to 2.0 should be not
> > > performance but some important features (like API or HA) that come
> > > with it.
> > >
> > > I couldn't agree more :). If we can start people writing (close to) 2.0
> > > > compliant DAGs before the release of 2.0 that will make the migration
> > > > process so much easier :).
> > > >
> > >
> > > Yeah.  I even thought that we should write a
> > > "How good your DAGs are for 2.0" assessment tool.
> > >
> > >
> > > > If there aren't any extra steps or features that we need to add (beyond
> > > the
> > > > ones discussed here), I think a good next step would be to create an
> > > > official checklist just so we can see all of these features in one
> > place
> > > > (and hopefully start breaking them down into as small of changes as
> > > > possible).
> > > >
> > > > Does that sound ok?
> > > >
> > >
> > > Perfectly OK for me!
> > >
> > >
> > > >
> > > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> >

Re: Let's talk Airflow 2.0

Posted by Robin Edwards <ro...@bidnamic.com>.

I feel some of the stuff for instance Schedular HA could wait for a point
release of version 2  (although maybe this a lot further a long than I am
aware). Like you mentioned Spark did with K8s.

Also does the new API need to be feature complete or just enough
functionality to warrant removing the existing experimental one.

Like you said, less sooner is better.

R

On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <da...@gmail.com>
wrote:

> Great! Hope to get a few more folx to give +1's but I think we have a good
> path forward here :)
>
> On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > >
> > >
> > > I agree especially for larger-scale users migrations are a difficult
> > > process. Perhaps we can adopt something similar to a blockchain fork
> > (e.g.
> > > determine X known airflow using companies, and start the countdown as
> > soon
> > > as Y% of them migrate). I really just want to make sure we don't end up
> > > with a python2/3 situation. Even if we continue support it should only
> be
> > > for bugfixes and we should not add any new features into 1.10.
> > >
> >
> > I think we are in perfect sync  - I think feature-migration should end
> > almost immediately after we release 2.0. But bug-fixing should continue
> > for quite some time. On that front - having backport packages will help
> > with releasing "integrations" quite independently from 1.10/2.0 version
> > (which I think is good for those who are - for this or another reason -
> > stuck on 2.0). On the other hand we should make sure that the important
> > stuff for 2.0 that is not "feature" is also backported to 1.10. For
> example
> > a lot of recent performance improvements that we have now in 2.0 will
> > be possible (and not that complex) to backport to 1.10. Some of this
> effort
> > is actually easier to do in 2.0 and then apply to 1.10 in similar fashion
> > as it is easier to understand and reason about the 2.0 code now when
> > we have some refactoring/pylints etc in place. So we should make sure
> > we get the latest 1.10 to a "good" state - before we freeze it for bugfix
> > only.
> > I know it might mean that some people will stay with 1.10 for longer, but
> > that's also OK for them. The reason to migrate to 2.0 should be not
> > performance but some important features (like API or HA) that come
> > with it.
> >
> > I couldn't agree more :). If we can start people writing (close to) 2.0
> > > compliant DAGs before the release of 2.0 that will make the migration
> > > process so much easier :).
> > >
> >
> > Yeah.  I even thought that we should write a
> > "How good your DAGs are for 2.0" assessment tool.
> >
> >
> > > If there aren't any extra steps or features that we need to add (beyond
> > the
> > > ones discussed here), I think a good next step would be to create an
> > > official checklist just so we can see all of these features in one
> place
> > > (and hopefully start breaking them down into as small of changes as
> > > possible).
> > >
> > > Does that sound ok?
> > >
> >
> > Perfectly OK for me!
> >
> >
> > >
> > > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>

Re: Let's talk Airflow 2.0

Posted by Daniel Imberman <da...@gmail.com>.

Great! Hope to get a few more folx to give +1's but I think we have a good
path forward here :)

On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <Ja...@polidea.com>
wrote:

> >
> >
> > I agree especially for larger-scale users migrations are a difficult
> > process. Perhaps we can adopt something similar to a blockchain fork
> (e.g.
> > determine X known airflow using companies, and start the countdown as
> soon
> > as Y% of them migrate). I really just want to make sure we don't end up
> > with a python2/3 situation. Even if we continue support it should only be
> > for bugfixes and we should not add any new features into 1.10.
> >
>
> I think we are in perfect sync  - I think feature-migration should end
> almost immediately after we release 2.0. But bug-fixing should continue
> for quite some time. On that front - having backport packages will help
> with releasing "integrations" quite independently from 1.10/2.0 version
> (which I think is good for those who are - for this or another reason -
> stuck on 2.0). On the other hand we should make sure that the important
> stuff for 2.0 that is not "feature" is also backported to 1.10. For example
> a lot of recent performance improvements that we have now in 2.0 will
> be possible (and not that complex) to backport to 1.10. Some of this effort
> is actually easier to do in 2.0 and then apply to 1.10 in similar fashion
> as it is easier to understand and reason about the 2.0 code now when
> we have some refactoring/pylints etc in place. So we should make sure
> we get the latest 1.10 to a "good" state - before we freeze it for bugfix
> only.
> I know it might mean that some people will stay with 1.10 for longer, but
> that's also OK for them. The reason to migrate to 2.0 should be not
> performance but some important features (like API or HA) that come
> with it.
>
> I couldn't agree more :). If we can start people writing (close to) 2.0
> > compliant DAGs before the release of 2.0 that will make the migration
> > process so much easier :).
> >
>
> Yeah.  I even thought that we should write a
> "How good your DAGs are for 2.0" assessment tool.
>
>
> > If there aren't any extra steps or features that we need to add (beyond
> the
> > ones discussed here), I think a good next step would be to create an
> > official checklist just so we can see all of these features in one place
> > (and hopefully start breaking them down into as small of changes as
> > possible).
> >
> > Does that sound ok?
> >
>
> Perfectly OK for me!
>
>
> >
> > --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>

Re: Let's talk Airflow 2.0

Posted by Jarek Potiuk <Ja...@polidea.com>.

>
>
> I agree especially for larger-scale users migrations are a difficult
> process. Perhaps we can adopt something similar to a blockchain fork (e.g.
> determine X known airflow using companies, and start the countdown as soon
> as Y% of them migrate). I really just want to make sure we don't end up
> with a python2/3 situation. Even if we continue support it should only be
> for bugfixes and we should not add any new features into 1.10.
>

I think we are in perfect sync  - I think feature-migration should end
almost immediately after we release 2.0. But bug-fixing should continue
for quite some time. On that front - having backport packages will help
with releasing "integrations" quite independently from 1.10/2.0 version
(which I think is good for those who are - for this or another reason -
stuck on 2.0). On the other hand we should make sure that the important
stuff for 2.0 that is not "feature" is also backported to 1.10. For example
a lot of recent performance improvements that we have now in 2.0 will
be possible (and not that complex) to backport to 1.10. Some of this effort
is actually easier to do in 2.0 and then apply to 1.10 in similar fashion
as it is easier to understand and reason about the 2.0 code now when
we have some refactoring/pylints etc in place. So we should make sure
we get the latest 1.10 to a "good" state - before we freeze it for bugfix
only.
I know it might mean that some people will stay with 1.10 for longer, but
that's also OK for them. The reason to migrate to 2.0 should be not
performance but some important features (like API or HA) that come
with it.

I couldn't agree more :). If we can start people writing (close to) 2.0
> compliant DAGs before the release of 2.0 that will make the migration
> process so much easier :).
>

Yeah.  I even thought that we should write a
"How good your DAGs are for 2.0" assessment tool.


> If there aren't any extra steps or features that we need to add (beyond the
> ones discussed here), I think a good next step would be to create an
> official checklist just so we can see all of these features in one place
> (and hopefully start breaking them down into as small of changes as
> possible).
>
> Does that sound ok?
>

Perfectly OK for me!


>
> --

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>

Re: Let's talk Airflow 2.0

Posted by Daniel Imberman <da...@gmail.com>.

Hi Jarek, thank you for your support on this effort!


For sure. One comment here. One of the ways we discussed with Kamil
> about the approach we want to take is to engage more people in
> implementing it - so Kamil would create a basic framework and we might
> have many people implementing and testing each endpoint separately.
> With Kaxil we are co-mentors in the Outreachyprogram and Airflow will
> have an intern paid by Outreachy helping with API development.
> Few Outreachy candidates recently started to contribute small tasks and
> we are going to select the intern at the beginning of April.


I think that's a fantastic idea. The more we can delegate and isolate tasks
the quicker we can move on this. Once we finish the migration to github
issues I'll be glad to help write up smaller tickets for each endpoint

We should provide some tools to migrate automatically (if possible) and
> asses the effort required to migrate and base our decision on tha
>

I think automatic migration tools are absolutely necessary and we can even
write system tests around those tools/use them for simultaneously testing
1.10 and 2.0

Of course, we should make our decision on our own based
> on what we hear, but I think 2 months for some of the Airflow users is not
> even
> enough time to start planning the migration. I'd say 6-12 months feels much
> more appropriate, but that's just feeling for now.
>

I agree especially for larger-scale users migrations are a difficult
process. Perhaps we can adopt something similar to a blockchain fork (e.g.
determine X known airflow using companies, and start the countdown as soon
as Y% of them migrate). I really just want to make sure we don't end up
with a python2/3 situation. Even if we continue support it should only be
for bugfixes and we should not add any new features into 1.10.

Absolutely. I think the system tests approach i kicked-off today might be
> the
> best way to ensure that the 2.0 "providers" part is battle-tested way
> before
> we release 2.0. This might make the whole migration process much
> smoother and faster.
>

I couldn't agree more :). If we can start people writing (close to) 2.0
compliant DAGs before the release of 2.0 that will make the migration
process so much easier :).

If there aren't any extra steps or features that we need to add (beyond the
ones discussed here), I think a good next step would be to create an
official checklist just so we can see all of these features in one place
(and hopefully start breaking them down into as small of changes as
possible).

Does that sound ok?

On Fri, Mar 20, 2020 at 9:57 AM Jarek Potiuk <Ja...@polidea.com>
wrote:

> > Hello, fellow Airflowers!
> >
> > I wanted to start this discussion in hopes of "kicking the tires" and
> > agreeing on a timeline for the Airflow 2.0 release.
>
> Cool!
>
> >
> > But I want to add _____ feature which would be so awesome in a 2.0 press
> > release!
> >
> > We should also
> > consider that the 2.0 release is ALREADY going to take a monumental
> testing
> > effort, and adding more features will only make that worse.
>
> Very true.
>
> > So what should we do now?
> >
> > Spring Cleaning of Necessary 2.0 tasks
> >
> > A while back, we created a spring cleaning list of 2.0 necessary fixes.
> > Let's review, triage the tickets no longer relevant, and give the highest
> > priority to the remaining tickets.
>
> Happy to help here.
>
> > Determine Breaking Changes
> >
> > Of all of the significant efforts in play right now, let's determine
> which
> > of those efforts are breaking. A major version bump is the only time we
> can
> > reasonably expect users to accept modifying DAGs, so let's categorize
> those
> > issues and prioritize them.
>
> I think a lot of that is already in UPDATING.md. But w will likely find
> more.
>
> > Release API and Scheduler HA
> >
> > Of the remaining efforts that could potentially cause breaking changes, I
> > believe that the new API, which replaces the “Experimental API”, and the
> > Scheduler HA have the most potential upside for users. Let's make sure to
> > include these in our planning for releasing 2.0.
>
> For sure. One comment here. One of the ways we discussed with Kamil
> about the approach we want to take is to engage more people in
> implementing it - so Kamil would create a basic framework and we might
> have many people implementing and testing each endpoint separately.
> With Kaxil we are co-mentors in the Outreachyprogram and Airflow will
> have an intern paid by Outreachy helping with API development.
> Few Outreachy candidates recently started to contribute small tasks and
> we are going to select the intern at the beginning of April.
>
> > Set a Timeline for 1.10 releases
> >
> > One question users will ask when this release comes out is, "how long
> will
> > the 1.10.x release stream continue to be enhanced?" In creating a
> timeline,
> > we want to ensure that customers are not left high and dry while also
> > preventing any form of Python 2/3 issue where users feel no urgency to
> > update. I propose a 2 month window where the 1.10.x release stream will
> > continue getting critical bug fixes, but not any additional features.
>
> I think that the decision on that should depend on the difficulty of
> migration.
> We should provide some tools to migrate automatically (if possible) and
> asses the effort required to migrate and base our decision on that (maybe
> we should also query our biggest users what their plans are for migration
> after
> we provide that. Many of our users are corporate users and we know it takes
> time to migrate. Of course, we should make our decision on our own based
> on what we hear, but I think 2 months for some of the Airflow users is not
> even
> enough time to start planning the migration. I'd say 6-12 months feels much
> more appropriate, but that's just feeling for now. Some survey might be a
> good way to check at least what are the expectations of our users.
>
> >
> > Enhance Tests to Avoid Regressions
> >
> > Let’s make sure that any new commits have tests suitable for both streams
> > while we get ready for 2.0. Let's continue our work on writing system
> > tests. Let’s all contribute scenarios so that we can ensure that these
> are
> > covered before we launch 2.0.
>
> Absolutely. I think the system tests approach i kicked-off today might be
> the
> best way to ensure that the 2.0 "providers" part is battle-tested way
> before
> we release 2.0. This might make the whole migration process much
> smoother and faster. I am looking forward to help from the community there
> :)
>
> > I'm ecstatic for the next year in Apache Airflow. Everything is speeding
> > up, and by cutting a 2.0 release, I think we can ensure that we keep up
> > with the pace of our growing community and demand.
>
> Yeah!
>
> J
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>

Re: Let's talk Airflow 2.0

Posted by Jarek Potiuk <Ja...@polidea.com>.

> Hello, fellow Airflowers!
>
> I wanted to start this discussion in hopes of "kicking the tires" and
> agreeing on a timeline for the Airflow 2.0 release.

Cool!

>
> But I want to add _____ feature which would be so awesome in a 2.0 press
> release!
>
> We should also
> consider that the 2.0 release is ALREADY going to take a monumental testing
> effort, and adding more features will only make that worse.

Very true.

> So what should we do now?
>
> Spring Cleaning of Necessary 2.0 tasks
>
> A while back, we created a spring cleaning list of 2.0 necessary fixes.
> Let's review, triage the tickets no longer relevant, and give the highest
> priority to the remaining tickets.

Happy to help here.

> Determine Breaking Changes
>
> Of all of the significant efforts in play right now, let's determine which
> of those efforts are breaking. A major version bump is the only time we can
> reasonably expect users to accept modifying DAGs, so let's categorize those
> issues and prioritize them.

I think a lot of that is already in UPDATING.md. But w will likely find more.

> Release API and Scheduler HA
>
> Of the remaining efforts that could potentially cause breaking changes, I
> believe that the new API, which replaces the “Experimental API”, and the
> Scheduler HA have the most potential upside for users. Let's make sure to
> include these in our planning for releasing 2.0.

For sure. One comment here. One of the ways we discussed with Kamil
about the approach we want to take is to engage more people in
implementing it - so Kamil would create a basic framework and we might
have many people implementing and testing each endpoint separately.
With Kaxil we are co-mentors in the Outreachyprogram and Airflow will
have an intern paid by Outreachy helping with API development.
Few Outreachy candidates recently started to contribute small tasks and
we are going to select the intern at the beginning of April.

> Set a Timeline for 1.10 releases
>
> One question users will ask when this release comes out is, "how long will
> the 1.10.x release stream continue to be enhanced?" In creating a timeline,
> we want to ensure that customers are not left high and dry while also
> preventing any form of Python 2/3 issue where users feel no urgency to
> update. I propose a 2 month window where the 1.10.x release stream will
> continue getting critical bug fixes, but not any additional features.

I think that the decision on that should depend on the difficulty of migration.
We should provide some tools to migrate automatically (if possible) and
asses the effort required to migrate and base our decision on that (maybe
we should also query our biggest users what their plans are for migration after
we provide that. Many of our users are corporate users and we know it takes
time to migrate. Of course, we should make our decision on our own based
on what we hear, but I think 2 months for some of the Airflow users is not even
enough time to start planning the migration. I'd say 6-12 months feels much
more appropriate, but that's just feeling for now. Some survey might be a
good way to check at least what are the expectations of our users.

>
> Enhance Tests to Avoid Regressions
>
> Let’s make sure that any new commits have tests suitable for both streams
> while we get ready for 2.0. Let's continue our work on writing system
> tests. Let’s all contribute scenarios so that we can ensure that these are
> covered before we launch 2.0.

Absolutely. I think the system tests approach i kicked-off today might be the
best way to ensure that the 2.0 "providers" part is battle-tested way before
we release 2.0. This might make the whole migration process much
smoother and faster. I am looking forward to help from the community there :)

> I'm ecstatic for the next year in Apache Airflow. Everything is speeding
> up, and by cutting a 2.0 release, I think we can ensure that we keep up
> with the pace of our growing community and demand.

Yeah!

J

-- 

Jarek Potiuk
Polidea | Principal Software Engineer

M: +48 660 796 129