You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2023/07/12 15:05:54 UTC
[DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Hello Everyone,
A small follow up after K8S/Celery executors being moved:
https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
We are in the process of moving Celery / Kubernetes executor (Celery almost
complete and I am working on K8S next + some common discovery and config
moving)
But there is one more "questionable" executor - i.e. Dask executor, still
living in Airflow Core.
When it comes to Celery/Kubernetes, we decided to make the two providers
preinstalled, because it makes most sense - we are also going to get the
basic documentation in the "core" airflow documentation so that it is
easier discoverable and prominently visible - also because of the
vendor-neutrality.
However when it comes to Dask I am not sure about its status and whether we
should make it preinstalled ?
I guess there is no doubt to move it to a provider - this has only the
benefits same as Celery/K8S move. But whether it should be preinstalled
with Airflow - I am not sure. I do not know how frequently Dask executor
(and Dask) is used by people using Airflow, but I personally do not think
it should be as "closely" connected with Airflow as Celery/Kubernetes ones.
If we do not make it preinstalled, it is somewhat (but not too much,
really) breaking change. We still might choose to install dask provider in
the PROD reference image, so it will continue to work if you use the image,
and when you are installing airflow in venv you will only have to specify
`pip install apache-airflow[dask]` or manually install
`apache-airflow-providers-daskexecutor` (for now at least this is the name
I could reserve in PyPI). So this is not really breaking, it just requires
another dependency to be installed. But some pipelines of installing
Airflow might get broken because it won't be pre-installed - so this is a
borderline breaking.
WDYT? Should we make the dask executor pre-installed or not?
J.
Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Posted by "Beck, Vincent" <vi...@amazon.com.INVALID>.
My opinion is I would rather not preinstall Dask provider because as already mentioned in this thread, this is not technically a breaking change. Users using Dask provider will be able to find out quite easily that their environment is not working without Dask dependency. To me it makes more sense to not have it preinstalled by default and only if needed, users just need to add it to their dependencies.
I also feel like (I might be wrong) this decision might take precedence on all the future similar decisions (whether provider X should be preinstalled by default in Airflow after having moved out the code from core to providers) and I feel, on the long term, the general "good" approach is to let users install the dependencies/providers they want without preinstalling them. I am not saying this is true for all providers, the ones that are essential to Airflow should, of course, be preinstalled. But this is not the case with Dask I think.
On 2023-07-20, 3:59 PM, "Daniel Standish" <daniel.standish@astronomer.io.INVA <ma...@astronomer.io.INVA>LID> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> I guess the question is not "can we do it" but more "should we do it" :D
right :)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
> I guess the question is not "can we do it" but more "should we do it" :D
right :)
Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Posted by Jarek Potiuk <ja...@potiuk.com>.
Yes. I quite agree.
We discussed it that "dependencies" are not "bound" by semver. And I think
technically we **could** make `dask` non-preinstalled without breaking
Semver. Similarly we could - and maybe even will (after AIP-56) have FAB as
an optional provider.
For me this is really the way how we could generally handle separating out
things and moving them "away" from Airflow's core without actually having
to bump Airflow to 3 (which I think would be a major marketing issue if we
start announcing "breaking" Airflow version.
We already have some precedent for that - I moved out Hive Macros that were
in "airflow" but now they are part of the Hive provider (and we
have PEP-562 dynamic attribute access with deprecation warnings in the core
- which is super small and 0-maintenance). So we have already done that.
And we could apply it to other things. For example if we would like to
remove SubDag support from core without bumping Airflow to 3, we could do
it. We just would need to find an easy way to extract it and make an API
that will allow an external provider to add a "subdag" feature and move it
out of the core. Not sure if it is feasible because SubDag is quite
"deeply" integrated in Airflow, but I think we could work it out and it
could be a great way to keep airflow core "lean" and move faster in general.
I guess the question is not "can we do it" but more "should we do it" :D
J.
On Thu, Jul 20, 2023 at 9:15 PM Daniel Standish
<da...@astronomer.io.invalid> wrote:
> Just on the question of semver, I am not convinced that semver prohibits
> this.
>
> As a user, your concerns and expectations regarding semver are about
> essentially how the code works, e.g. are you going to have to refactor all
> of your 500 dags. In other words the API.. But to me this is a lower
> level thing that doesn't seem should strictly be forbidden. It's like...
> when we change the dependencies in airflow, inevitably that will break
> someone's cluster if they blindly deploy to prod. But this is not a
> prudent thing to do. So I think it's not an unreasonable expectation that a
> user verify that their cluster will spin up. It's something you can verify
> without running any dags at all (i.e. if you set dask executor and you
> start airflow, i think airflow will not start). And then users can rely on
> semver about changing behavior, which is not something you necessarily can
> tell without running dags.
>
Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
Just on the question of semver, I am not convinced that semver prohibits
this.
As a user, your concerns and expectations regarding semver are about
essentially how the code works, e.g. are you going to have to refactor all
of your 500 dags. In other words the API.. But to me this is a lower
level thing that doesn't seem should strictly be forbidden. It's like...
when we change the dependencies in airflow, inevitably that will break
someone's cluster if they blindly deploy to prod. But this is not a
prudent thing to do. So I think it's not an unreasonable expectation that a
user verify that their cluster will spin up. It's something you can verify
without running any dags at all (i.e. if you set dask executor and you
start airflow, i think airflow will not start). And then users can rely on
semver about changing behavior, which is not something you necessarily can
tell without running dags.
Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Posted by Jarek Potiuk <ja...@potiuk.com>.
Ok. I am ok with leaving Dask preinstalled, I think we will already benefit
from moving it to provider and moving tests to the provider tests.
On Fri, Jul 14, 2023 at 3:08 PM Hussein Awala <hu...@awala.fr> wrote:
> We all agree on moving the 5 remote executors to their respective
> providers. The main question now is whether we should pre-install all the
> providers or not.
>
> IMHO we should pre-install Celery, Kubernetes, and Dask providers until
> Airflow 3.0. Not doing so would be a breaking change for some users, and it
> would go against the Airflow deprecation policy.
>
> Once Airflow 3.0 is released, we should remove the pre-installed providers
> and instead prompt the user to install the provider specific to its
> executor.
>
> On Fri 14 Jul 2023 at 14:54, utkarsh sharma <ut...@gmail.com>
> wrote:
>
> > I'm too in favor of moving Dask Executor into its own provider, which
> will
> > make the airflow's codebase more pluggable and orthogonal. Big +1 :)
> >
> > Thanks,
> > Utkarsh Sharma
> >
> > On Fri, Jul 14, 2023 at 2:03 AM Ferruzzi, Dennis
> > <fe...@amazon.com.invalid> wrote:
> >
> > > I'm not sure how much of a train wreck it would turn into, but moving
> > Dask
> > > Executor to a provider seems logical to me.
> > >
> > > Maybe in the far-flung future of Airflow 3.0 we could move celery and
> k8s
> > > into their own as well and make it truly pluggable and
> executor-agnostic,
> > > but I agree that at this time they are too deeply integrated.
> > >
> > >
> > > - ferruzzi
> > >
> > >
> > > ________________________________
> > > From: Oliveira, Niko <on...@amazon.com.INVALID>
> > > Sent: Wednesday, July 12, 2023 10:40 AM
> > > To: dev@airflow.apache.org
> > > Subject: RE: [EXTERNAL][DISCUSS] Moving Dask Executor to a separate
> > > (optional?) dask provider
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > > click links or open attachments unless you can confirm the sender and
> > know
> > > the content is safe.
> > >
> > >
> > >
> > > I think in a perfect world we'd only have the completely vendor neutral
> > > executors pre-installed (Local, Sequential, Debug) and anything else
> > would
> > > need to be specifically installed by admins/users. I think if we were
> > > starting from scratch this would make the most sense, but clearly
> > > Kubernetes and Celery executors are so ubiquitous that it'd cause too
> > much
> > > wreckage to not install them, but I'd like to push for Dask to _not_ be
> > > installed by default. If this causes too much wreckage then perhaps we
> > > should deprecate that (though I'm not sure exactly what that would look
> > > like in this context), but it's difficult to measure how many folks are
> > > using the Dask executor. Perhaps we have data from the yearly
> > > questionnaire/survey we send?
> > >
> > > ________________________________
> > > From: Jarek Potiuk <ja...@potiuk.com>
> > > Sent: Wednesday, July 12, 2023 8:05:54 AM
> > > To: dev@airflow.apache.org
> > > Subject: [EXTERNAL] [DISCUSS] Moving Dask Executor to a separate
> > > (optional?) dask provider
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > > click links or open attachments unless you can confirm the sender and
> > know
> > > the content is safe.
> > >
> > >
> > >
> > > Hello Everyone,
> > >
> > > A small follow up after K8S/Celery executors being moved:
> > > https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
> > >
> > > We are in the process of moving Celery / Kubernetes executor (Celery
> > almost
> > > complete and I am working on K8S next + some common discovery and
> config
> > > moving)
> > >
> > > But there is one more "questionable" executor - i.e. Dask executor,
> still
> > > living in Airflow Core.
> > >
> > > When it comes to Celery/Kubernetes, we decided to make the two
> providers
> > > preinstalled, because it makes most sense - we are also going to get
> the
> > > basic documentation in the "core" airflow documentation so that it is
> > > easier discoverable and prominently visible - also because of the
> > > vendor-neutrality.
> > >
> > > However when it comes to Dask I am not sure about its status and
> whether
> > we
> > > should make it preinstalled ?
> > >
> > > I guess there is no doubt to move it to a provider - this has only the
> > > benefits same as Celery/K8S move. But whether it should be preinstalled
> > > with Airflow - I am not sure. I do not know how frequently Dask
> executor
> > > (and Dask) is used by people using Airflow, but I personally do not
> think
> > > it should be as "closely" connected with Airflow as Celery/Kubernetes
> > ones.
> > >
> > > If we do not make it preinstalled, it is somewhat (but not too much,
> > > really) breaking change. We still might choose to install dask provider
> > in
> > > the PROD reference image, so it will continue to work if you use the
> > image,
> > > and when you are installing airflow in venv you will only have to
> specify
> > > `pip install apache-airflow[dask]` or manually install
> > > `apache-airflow-providers-daskexecutor` (for now at least this is the
> > name
> > > I could reserve in PyPI). So this is not really breaking, it just
> > requires
> > > another dependency to be installed. But some pipelines of installing
> > > Airflow might get broken because it won't be pre-installed - so this
> is a
> > > borderline breaking.
> > >
> > > WDYT? Should we make the dask executor pre-installed or not?
> > >
> > > J.
> > >
> >
>
Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Posted by Hussein Awala <hu...@awala.fr>.
We all agree on moving the 5 remote executors to their respective
providers. The main question now is whether we should pre-install all the
providers or not.
IMHO we should pre-install Celery, Kubernetes, and Dask providers until
Airflow 3.0. Not doing so would be a breaking change for some users, and it
would go against the Airflow deprecation policy.
Once Airflow 3.0 is released, we should remove the pre-installed providers
and instead prompt the user to install the provider specific to its
executor.
On Fri 14 Jul 2023 at 14:54, utkarsh sharma <ut...@gmail.com> wrote:
> I'm too in favor of moving Dask Executor into its own provider, which will
> make the airflow's codebase more pluggable and orthogonal. Big +1 :)
>
> Thanks,
> Utkarsh Sharma
>
> On Fri, Jul 14, 2023 at 2:03 AM Ferruzzi, Dennis
> <fe...@amazon.com.invalid> wrote:
>
> > I'm not sure how much of a train wreck it would turn into, but moving
> Dask
> > Executor to a provider seems logical to me.
> >
> > Maybe in the far-flung future of Airflow 3.0 we could move celery and k8s
> > into their own as well and make it truly pluggable and executor-agnostic,
> > but I agree that at this time they are too deeply integrated.
> >
> >
> > - ferruzzi
> >
> >
> > ________________________________
> > From: Oliveira, Niko <on...@amazon.com.INVALID>
> > Sent: Wednesday, July 12, 2023 10:40 AM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXTERNAL][DISCUSS] Moving Dask Executor to a separate
> > (optional?) dask provider
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > I think in a perfect world we'd only have the completely vendor neutral
> > executors pre-installed (Local, Sequential, Debug) and anything else
> would
> > need to be specifically installed by admins/users. I think if we were
> > starting from scratch this would make the most sense, but clearly
> > Kubernetes and Celery executors are so ubiquitous that it'd cause too
> much
> > wreckage to not install them, but I'd like to push for Dask to _not_ be
> > installed by default. If this causes too much wreckage then perhaps we
> > should deprecate that (though I'm not sure exactly what that would look
> > like in this context), but it's difficult to measure how many folks are
> > using the Dask executor. Perhaps we have data from the yearly
> > questionnaire/survey we send?
> >
> > ________________________________
> > From: Jarek Potiuk <ja...@potiuk.com>
> > Sent: Wednesday, July 12, 2023 8:05:54 AM
> > To: dev@airflow.apache.org
> > Subject: [EXTERNAL] [DISCUSS] Moving Dask Executor to a separate
> > (optional?) dask provider
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > Hello Everyone,
> >
> > A small follow up after K8S/Celery executors being moved:
> > https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
> >
> > We are in the process of moving Celery / Kubernetes executor (Celery
> almost
> > complete and I am working on K8S next + some common discovery and config
> > moving)
> >
> > But there is one more "questionable" executor - i.e. Dask executor, still
> > living in Airflow Core.
> >
> > When it comes to Celery/Kubernetes, we decided to make the two providers
> > preinstalled, because it makes most sense - we are also going to get the
> > basic documentation in the "core" airflow documentation so that it is
> > easier discoverable and prominently visible - also because of the
> > vendor-neutrality.
> >
> > However when it comes to Dask I am not sure about its status and whether
> we
> > should make it preinstalled ?
> >
> > I guess there is no doubt to move it to a provider - this has only the
> > benefits same as Celery/K8S move. But whether it should be preinstalled
> > with Airflow - I am not sure. I do not know how frequently Dask executor
> > (and Dask) is used by people using Airflow, but I personally do not think
> > it should be as "closely" connected with Airflow as Celery/Kubernetes
> ones.
> >
> > If we do not make it preinstalled, it is somewhat (but not too much,
> > really) breaking change. We still might choose to install dask provider
> in
> > the PROD reference image, so it will continue to work if you use the
> image,
> > and when you are installing airflow in venv you will only have to specify
> > `pip install apache-airflow[dask]` or manually install
> > `apache-airflow-providers-daskexecutor` (for now at least this is the
> name
> > I could reserve in PyPI). So this is not really breaking, it just
> requires
> > another dependency to be installed. But some pipelines of installing
> > Airflow might get broken because it won't be pre-installed - so this is a
> > borderline breaking.
> >
> > WDYT? Should we make the dask executor pre-installed or not?
> >
> > J.
> >
>
Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Posted by utkarsh sharma <ut...@gmail.com>.
I'm too in favor of moving Dask Executor into its own provider, which will
make the airflow's codebase more pluggable and orthogonal. Big +1 :)
Thanks,
Utkarsh Sharma
On Fri, Jul 14, 2023 at 2:03 AM Ferruzzi, Dennis
<fe...@amazon.com.invalid> wrote:
> I'm not sure how much of a train wreck it would turn into, but moving Dask
> Executor to a provider seems logical to me.
>
> Maybe in the far-flung future of Airflow 3.0 we could move celery and k8s
> into their own as well and make it truly pluggable and executor-agnostic,
> but I agree that at this time they are too deeply integrated.
>
>
> - ferruzzi
>
>
> ________________________________
> From: Oliveira, Niko <on...@amazon.com.INVALID>
> Sent: Wednesday, July 12, 2023 10:40 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL][DISCUSS] Moving Dask Executor to a separate
> (optional?) dask provider
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> I think in a perfect world we'd only have the completely vendor neutral
> executors pre-installed (Local, Sequential, Debug) and anything else would
> need to be specifically installed by admins/users. I think if we were
> starting from scratch this would make the most sense, but clearly
> Kubernetes and Celery executors are so ubiquitous that it'd cause too much
> wreckage to not install them, but I'd like to push for Dask to _not_ be
> installed by default. If this causes too much wreckage then perhaps we
> should deprecate that (though I'm not sure exactly what that would look
> like in this context), but it's difficult to measure how many folks are
> using the Dask executor. Perhaps we have data from the yearly
> questionnaire/survey we send?
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Wednesday, July 12, 2023 8:05:54 AM
> To: dev@airflow.apache.org
> Subject: [EXTERNAL] [DISCUSS] Moving Dask Executor to a separate
> (optional?) dask provider
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hello Everyone,
>
> A small follow up after K8S/Celery executors being moved:
> https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
>
> We are in the process of moving Celery / Kubernetes executor (Celery almost
> complete and I am working on K8S next + some common discovery and config
> moving)
>
> But there is one more "questionable" executor - i.e. Dask executor, still
> living in Airflow Core.
>
> When it comes to Celery/Kubernetes, we decided to make the two providers
> preinstalled, because it makes most sense - we are also going to get the
> basic documentation in the "core" airflow documentation so that it is
> easier discoverable and prominently visible - also because of the
> vendor-neutrality.
>
> However when it comes to Dask I am not sure about its status and whether we
> should make it preinstalled ?
>
> I guess there is no doubt to move it to a provider - this has only the
> benefits same as Celery/K8S move. But whether it should be preinstalled
> with Airflow - I am not sure. I do not know how frequently Dask executor
> (and Dask) is used by people using Airflow, but I personally do not think
> it should be as "closely" connected with Airflow as Celery/Kubernetes ones.
>
> If we do not make it preinstalled, it is somewhat (but not too much,
> really) breaking change. We still might choose to install dask provider in
> the PROD reference image, so it will continue to work if you use the image,
> and when you are installing airflow in venv you will only have to specify
> `pip install apache-airflow[dask]` or manually install
> `apache-airflow-providers-daskexecutor` (for now at least this is the name
> I could reserve in PyPI). So this is not really breaking, it just requires
> another dependency to be installed. But some pipelines of installing
> Airflow might get broken because it won't be pre-installed - so this is a
> borderline breaking.
>
> WDYT? Should we make the dask executor pre-installed or not?
>
> J.
>
Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
I'm not sure how much of a train wreck it would turn into, but moving Dask Executor to a provider seems logical to me.
Maybe in the far-flung future of Airflow 3.0 we could move celery and k8s into their own as well and make it truly pluggable and executor-agnostic, but I agree that at this time they are too deeply integrated.
- ferruzzi
________________________________
From: Oliveira, Niko <on...@amazon.com.INVALID>
Sent: Wednesday, July 12, 2023 10:40 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL][DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
I think in a perfect world we'd only have the completely vendor neutral executors pre-installed (Local, Sequential, Debug) and anything else would need to be specifically installed by admins/users. I think if we were starting from scratch this would make the most sense, but clearly Kubernetes and Celery executors are so ubiquitous that it'd cause too much wreckage to not install them, but I'd like to push for Dask to _not_ be installed by default. If this causes too much wreckage then perhaps we should deprecate that (though I'm not sure exactly what that would look like in this context), but it's difficult to measure how many folks are using the Dask executor. Perhaps we have data from the yearly questionnaire/survey we send?
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Wednesday, July 12, 2023 8:05:54 AM
To: dev@airflow.apache.org
Subject: [EXTERNAL] [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hello Everyone,
A small follow up after K8S/Celery executors being moved:
https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
We are in the process of moving Celery / Kubernetes executor (Celery almost
complete and I am working on K8S next + some common discovery and config
moving)
But there is one more "questionable" executor - i.e. Dask executor, still
living in Airflow Core.
When it comes to Celery/Kubernetes, we decided to make the two providers
preinstalled, because it makes most sense - we are also going to get the
basic documentation in the "core" airflow documentation so that it is
easier discoverable and prominently visible - also because of the
vendor-neutrality.
However when it comes to Dask I am not sure about its status and whether we
should make it preinstalled ?
I guess there is no doubt to move it to a provider - this has only the
benefits same as Celery/K8S move. But whether it should be preinstalled
with Airflow - I am not sure. I do not know how frequently Dask executor
(and Dask) is used by people using Airflow, but I personally do not think
it should be as "closely" connected with Airflow as Celery/Kubernetes ones.
If we do not make it preinstalled, it is somewhat (but not too much,
really) breaking change. We still might choose to install dask provider in
the PROD reference image, so it will continue to work if you use the image,
and when you are installing airflow in venv you will only have to specify
`pip install apache-airflow[dask]` or manually install
`apache-airflow-providers-daskexecutor` (for now at least this is the name
I could reserve in PyPI). So this is not really breaking, it just requires
another dependency to be installed. But some pipelines of installing
Airflow might get broken because it won't be pre-installed - so this is a
borderline breaking.
WDYT? Should we make the dask executor pre-installed or not?
J.
Re: [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
Posted by "Oliveira, Niko" <on...@amazon.com.INVALID>.
I think in a perfect world we'd only have the completely vendor neutral executors pre-installed (Local, Sequential, Debug) and anything else would need to be specifically installed by admins/users. I think if we were starting from scratch this would make the most sense, but clearly Kubernetes and Celery executors are so ubiquitous that it'd cause too much wreckage to not install them, but I'd like to push for Dask to _not_ be installed by default. If this causes too much wreckage then perhaps we should deprecate that (though I'm not sure exactly what that would look like in this context), but it's difficult to measure how many folks are using the Dask executor. Perhaps we have data from the yearly questionnaire/survey we send?
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Wednesday, July 12, 2023 8:05:54 AM
To: dev@airflow.apache.org
Subject: [EXTERNAL] [DISCUSS] Moving Dask Executor to a separate (optional?) dask provider
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hello Everyone,
A small follow up after K8S/Celery executors being moved:
https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
We are in the process of moving Celery / Kubernetes executor (Celery almost
complete and I am working on K8S next + some common discovery and config
moving)
But there is one more "questionable" executor - i.e. Dask executor, still
living in Airflow Core.
When it comes to Celery/Kubernetes, we decided to make the two providers
preinstalled, because it makes most sense - we are also going to get the
basic documentation in the "core" airflow documentation so that it is
easier discoverable and prominently visible - also because of the
vendor-neutrality.
However when it comes to Dask I am not sure about its status and whether we
should make it preinstalled ?
I guess there is no doubt to move it to a provider - this has only the
benefits same as Celery/K8S move. But whether it should be preinstalled
with Airflow - I am not sure. I do not know how frequently Dask executor
(and Dask) is used by people using Airflow, but I personally do not think
it should be as "closely" connected with Airflow as Celery/Kubernetes ones.
If we do not make it preinstalled, it is somewhat (but not too much,
really) breaking change. We still might choose to install dask provider in
the PROD reference image, so it will continue to work if you use the image,
and when you are installing airflow in venv you will only have to specify
`pip install apache-airflow[dask]` or manually install
`apache-airflow-providers-daskexecutor` (for now at least this is the name
I could reserve in PyPI). So this is not really breaking, it just requires
another dependency to be installed. But some pipelines of installing
Airflow might get broken because it won't be pre-installed - so this is a
borderline breaking.
WDYT? Should we make the dask executor pre-installed or not?
J.