AIP-25 The Knative Executor

[Daniel Imberman <da...@gmail.com>]

Hello all!

Over the last two years, there has been a lot of incredible adoption and
feedback regarding the KubernetesExecutor. While there have been many
positive improvements within this feature, one thing that has been commonly
requested is a better story around scaling and start-up time. I (and some
extremely helpful colleagues) wanted to come up with a solution that could
both offer the dynamic allocation of the KubernetesExecutor with the
scale/start-up time improvements of having shared airflow machines. With
these constraints in mind, I want to propose a Knative Executor.

I've written up an AIP for the wiki below

https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-25+The+Knative+Executor

But for early edits I would like to offer this google doc link for early
edits (the google doc will allow for in-line comments and edit suggestions).

https://docs.google.com/document/d/1xcSjkYv9Sh1Vavb-tvLUdy-8AHetrWKfiIblQE1rdYc/edit?usp=sharing

I'm really excited to get feedback on this and hope that this is the part
of an exciting new chapter in the airflow + kubernetes integration :).

Thank you all for your consideration on this AIP!

Daniel

Re: AIP-25 The Knative Executor

[Daniel Imberman <da...@gmail.com>]

Also huge thank you to @James Meickle <jm...@quantopian.com> @Aizhamal
Nurmamat kyzy <ai...@google.com> @Ash Berlin-Taylor <as...@apache.org> and
the google knative team for really valuable initial feedback :).

On Fri, Aug 16, 2019 at 10:17 AM Daniel Imberman <da...@gmail.com>
wrote:

> Hello all!
>
> Over the last two years, there has been a lot of incredible adoption and
> feedback regarding the KubernetesExecutor. While there have been many
> positive improvements within this feature, one thing that has been commonly
> requested is a better story around scaling and start-up time. I (and some
> extremely helpful colleagues) wanted to come up with a solution that could
> both offer the dynamic allocation of the KubernetesExecutor with the
> scale/start-up time improvements of having shared airflow machines. With
> these constraints in mind, I want to propose a Knative Executor.
>
> I've written up an AIP for the wiki below
>
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-25+The+Knative+Executor
>
> But for early edits I would like to offer this google doc link for early
> edits (the google doc will allow for in-line comments and edit suggestions).
>
>
> https://docs.google.com/document/d/1xcSjkYv9Sh1Vavb-tvLUdy-8AHetrWKfiIblQE1rdYc/edit?usp=sharing
>
> I'm really excited to get feedback on this and hope that this is the part
> of an exciting new chapter in the airflow + kubernetes integration :).
>
> Thank you all for your consideration on this AIP!
>
> Daniel
>