[DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

Hello Everyone,

I would like to start discussion about potential suspension and eventually
removal of Daskexecutor provider.

After some recent changes and moving DaskExecutor from core to provider we
have now an open option to suspend and eventually remove Dask Executor from
our codebase.

After we have the process tested and tried with Qubole, for me this is the
next candidate to remove and get rid of some burden it involves.

I believe this executor is hardly used by anyone - IMHO it does not really
add a value to our Executor's offering and it causes a number of CI/
maintenance issues for us, while we have little to no experience and
incentive and knowledge to manage it. It slows us down, makes our CI flaky,
sometimes delays our release process etc etc.

Some of the past discussions about it (this is the third time I am (re)
starting this discussion - the first time it was in 2020 when we discussed
removing it for Airflow 2.0.

* https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m
* https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh

Some of the recent problems with Dask executor impacting stability of our
CI / tests/ impacting release process are manifested in those issues and
PRs over the last few years (and this is just a subset of issues and PRs
dealing with the stability of it.

* https://github.com/apache/airflow/pull/35659
* https://github.com/apache/airflow/issues/32778
* https://github.com/apache/airflow/pull/35473
* https://github.com/apache/airflow/pull/32991
* https://github.com/apache/airflow/pull/30258
* https://github.com/apache/airflow/pull/22046
* https://github.com/apache/airflow/pull/22017


Why now? What changed?

One of the fantastic properties of the "providers" approach and moving the
executor from core to provider - we can now ACTUALLY remove DaskExecutor
without breaking any of our SemVer promises. This is something I was
working on for quite a while and with some of the recent changes - making
our Executor API stable and part of our public interface, introducing full
life-cycle process for providers this is now entirely possible to remove
all the maintenance burden while keeping our SemVer promises - so that we
can do it without bumping Airflow to version 3.

How does it work ? Simply - provider will remain released, and it will
remain working in the version it has been released for the last time with
all the future versions of Airflow, But at the same time we can remove the
code from main of Airflow, stop running tests, stop caring about the
provider impacting the constraints and generally speaking.

If we decide to do somethin about, we have three options:

1) Suspend the daskexecutor provider
2) Remove it
3) We can also follow Suspend -> wait few months -> remove) pattern

All of them are easy, all of them bring us much welcome relief in our
release and CI and maintenance burden. All of them are well documented and
tried and all that we need is just to agree which scenario we want to
follow.

In case 1) we will stop releasing new versions, but in case any new
"development" is done on the dask side that will break currently released
providers, it will be on someone who will take on the task to bring it back
from suspension - fix all dependency issues, make sure tests are green and
then it's just a matter of PR to bring it back. This is really a way to say
"Hey Dask Executor community, if you want to keep any future development,
it's on you to bring it back from suspension and take the task to solve all
the difficulties we currently have to leave with in Airflow community".

In case 2) we deliberately decide we do not want to maintain it any more
and if someone would like to bring it back, they will have to convince the
community and get the VOTE or LAZY CONSENSUS to bring it back (and then it
could be even different provider package, not compatible etc.).

WDYT? Are there any DaskExecutor users around or someone you know of who
would rely on it heavily? Are there any reasons why we should keep it as is
? Which of the above scenarios would you like to see?

I would love to hear your opinions on that. If we see that we would like to
follow 1) or 2) I am happy to pass the message to the Dask community and
let them know that they will need to be involved in case they will want to
bring back Dask from suspension or removal - I did that before for some of
the past issues and I am happy to be the messenger (hopefull no-one shoots
me).

J.

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

Maintaining security fixes is independent from suspension/removal.

We are obliged to release security fixes regardless if it's something in
our code. For example if we have a report for Qubole and we assess it is
valid and it will need change in our code, we should still do it for quite
some time (we do not have specific expectations for that time - but in the
future CRA regulation there is an expectation to set " expected time of
life" for anything released and this would be as long as we set it then).

In both cases if we need to do it - we would do it branching off at the TAG
when the last release happened and develop/release a fix from there.

J.


On Wed, Nov 15, 2023 at 9:30 PM Hussein Awala <hu...@awala.fr> wrote:

> Since this executor was part of the Airflow core a few months ago, should
> we maintain security patches for a reasonable period after officially
> announcing its suspension?
>
> On Wed, Nov 15, 2023 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hello Everyone,
>>
>> I would like to start discussion about potential suspension and eventually
>> removal of Daskexecutor provider.
>>
>> After some recent changes and moving DaskExecutor from core to provider we
>> have now an open option to suspend and eventually remove Dask Executor
>> from
>> our codebase.
>>
>> After we have the process tested and tried with Qubole, for me this is the
>> next candidate to remove and get rid of some burden it involves.
>>
>> I believe this executor is hardly used by anyone - IMHO it does not really
>> add a value to our Executor's offering and it causes a number of CI/
>> maintenance issues for us, while we have little to no experience and
>> incentive and knowledge to manage it. It slows us down, makes our CI
>> flaky,
>> sometimes delays our release process etc etc.
>>
>> Some of the past discussions about it (this is the third time I am (re)
>> starting this discussion - the first time it was in 2020 when we discussed
>> removing it for Airflow 2.0.
>>
>> * https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m
>> * https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh
>>
>> Some of the recent problems with Dask executor impacting stability of our
>> CI / tests/ impacting release process are manifested in those issues and
>> PRs over the last few years (and this is just a subset of issues and PRs
>> dealing with the stability of it.
>>
>> * https://github.com/apache/airflow/pull/35659
>> * https://github.com/apache/airflow/issues/32778
>> * https://github.com/apache/airflow/pull/35473
>> * https://github.com/apache/airflow/pull/32991
>> * https://github.com/apache/airflow/pull/30258
>> * https://github.com/apache/airflow/pull/22046
>> * https://github.com/apache/airflow/pull/22017
>>
>>
>> Why now? What changed?
>>
>> One of the fantastic properties of the "providers" approach and moving the
>> executor from core to provider - we can now ACTUALLY remove DaskExecutor
>> without breaking any of our SemVer promises. This is something I was
>> working on for quite a while and with some of the recent changes - making
>> our Executor API stable and part of our public interface, introducing full
>> life-cycle process for providers this is now entirely possible to remove
>> all the maintenance burden while keeping our SemVer promises - so that we
>> can do it without bumping Airflow to version 3.
>>
>> How does it work ? Simply - provider will remain released, and it will
>> remain working in the version it has been released for the last time with
>> all the future versions of Airflow, But at the same time we can remove the
>> code from main of Airflow, stop running tests, stop caring about the
>> provider impacting the constraints and generally speaking.
>>
>> If we decide to do somethin about, we have three options:
>>
>> 1) Suspend the daskexecutor provider
>> 2) Remove it
>> 3) We can also follow Suspend -> wait few months -> remove) pattern
>>
>> All of them are easy, all of them bring us much welcome relief in our
>> release and CI and maintenance burden. All of them are well documented and
>> tried and all that we need is just to agree which scenario we want to
>> follow.
>>
>> In case 1) we will stop releasing new versions, but in case any new
>> "development" is done on the dask side that will break currently released
>> providers, it will be on someone who will take on the task to bring it
>> back
>> from suspension - fix all dependency issues, make sure tests are green and
>> then it's just a matter of PR to bring it back. This is really a way to
>> say
>> "Hey Dask Executor community, if you want to keep any future development,
>> it's on you to bring it back from suspension and take the task to solve
>> all
>> the difficulties we currently have to leave with in Airflow community".
>>
>> In case 2) we deliberately decide we do not want to maintain it any more
>> and if someone would like to bring it back, they will have to convince the
>> community and get the VOTE or LAZY CONSENSUS to bring it back (and then it
>> could be even different provider package, not compatible etc.).
>>
>> WDYT? Are there any DaskExecutor users around or someone you know of who
>> would rely on it heavily? Are there any reasons why we should keep it as
>> is
>> ? Which of the above scenarios would you like to see?
>>
>> I would love to hear your opinions on that. If we see that we would like
>> to
>> follow 1) or 2) I am happy to pass the message to the Dask community and
>> let them know that they will need to be involved in case they will want to
>> bring back Dask from suspension or removal - I did that before for some of
>> the past issues and I am happy to be the messenger (hopefull no-one shoots
>> me).
>>
>> J.
>>
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

Maintaining security fixes is independent from suspension/removal.

We are obliged to release security fixes regardless if it's something in
our code. For example if we have a report for Qubole and we assess it is
valid and it will need change in our code, we should still do it for quite
some time (we do not have specific expectations for that time - but in the
future CRA regulation there is an expectation to set " expected time of
life" for anything released and this would be as long as we set it then).

In both cases if we need to do it - we would do it branching off at the TAG
when the last release happened and develop/release a fix from there.

J.


On Wed, Nov 15, 2023 at 9:30 PM Hussein Awala <hu...@awala.fr> wrote:

> Since this executor was part of the Airflow core a few months ago, should
> we maintain security patches for a reasonable period after officially
> announcing its suspension?
>
> On Wed, Nov 15, 2023 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Hello Everyone,
>>
>> I would like to start discussion about potential suspension and eventually
>> removal of Daskexecutor provider.
>>
>> After some recent changes and moving DaskExecutor from core to provider we
>> have now an open option to suspend and eventually remove Dask Executor
>> from
>> our codebase.
>>
>> After we have the process tested and tried with Qubole, for me this is the
>> next candidate to remove and get rid of some burden it involves.
>>
>> I believe this executor is hardly used by anyone - IMHO it does not really
>> add a value to our Executor's offering and it causes a number of CI/
>> maintenance issues for us, while we have little to no experience and
>> incentive and knowledge to manage it. It slows us down, makes our CI
>> flaky,
>> sometimes delays our release process etc etc.
>>
>> Some of the past discussions about it (this is the third time I am (re)
>> starting this discussion - the first time it was in 2020 when we discussed
>> removing it for Airflow 2.0.
>>
>> * https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m
>> * https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh
>>
>> Some of the recent problems with Dask executor impacting stability of our
>> CI / tests/ impacting release process are manifested in those issues and
>> PRs over the last few years (and this is just a subset of issues and PRs
>> dealing with the stability of it.
>>
>> * https://github.com/apache/airflow/pull/35659
>> * https://github.com/apache/airflow/issues/32778
>> * https://github.com/apache/airflow/pull/35473
>> * https://github.com/apache/airflow/pull/32991
>> * https://github.com/apache/airflow/pull/30258
>> * https://github.com/apache/airflow/pull/22046
>> * https://github.com/apache/airflow/pull/22017
>>
>>
>> Why now? What changed?
>>
>> One of the fantastic properties of the "providers" approach and moving the
>> executor from core to provider - we can now ACTUALLY remove DaskExecutor
>> without breaking any of our SemVer promises. This is something I was
>> working on for quite a while and with some of the recent changes - making
>> our Executor API stable and part of our public interface, introducing full
>> life-cycle process for providers this is now entirely possible to remove
>> all the maintenance burden while keeping our SemVer promises - so that we
>> can do it without bumping Airflow to version 3.
>>
>> How does it work ? Simply - provider will remain released, and it will
>> remain working in the version it has been released for the last time with
>> all the future versions of Airflow, But at the same time we can remove the
>> code from main of Airflow, stop running tests, stop caring about the
>> provider impacting the constraints and generally speaking.
>>
>> If we decide to do somethin about, we have three options:
>>
>> 1) Suspend the daskexecutor provider
>> 2) Remove it
>> 3) We can also follow Suspend -> wait few months -> remove) pattern
>>
>> All of them are easy, all of them bring us much welcome relief in our
>> release and CI and maintenance burden. All of them are well documented and
>> tried and all that we need is just to agree which scenario we want to
>> follow.
>>
>> In case 1) we will stop releasing new versions, but in case any new
>> "development" is done on the dask side that will break currently released
>> providers, it will be on someone who will take on the task to bring it
>> back
>> from suspension - fix all dependency issues, make sure tests are green and
>> then it's just a matter of PR to bring it back. This is really a way to
>> say
>> "Hey Dask Executor community, if you want to keep any future development,
>> it's on you to bring it back from suspension and take the task to solve
>> all
>> the difficulties we currently have to leave with in Airflow community".
>>
>> In case 2) we deliberately decide we do not want to maintain it any more
>> and if someone would like to bring it back, they will have to convince the
>> community and get the VOTE or LAZY CONSENSUS to bring it back (and then it
>> could be even different provider package, not compatible etc.).
>>
>> WDYT? Are there any DaskExecutor users around or someone you know of who
>> would rely on it heavily? Are there any reasons why we should keep it as
>> is
>> ? Which of the above scenarios would you like to see?
>>
>> I would love to hear your opinions on that. If we see that we would like
>> to
>> follow 1) or 2) I am happy to pass the message to the Dask community and
>> let them know that they will need to be involved in case they will want to
>> bring back Dask from suspension or removal - I did that before for some of
>> the past issues and I am happy to be the messenger (hopefull no-one shoots
>> me).
>>
>> J.
>>
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Hussein Awala <hu...@awala.fr>]

Since this executor was part of the Airflow core a few months ago, should
we maintain security patches for a reasonable period after officially
announcing its suspension?

On Wed, Nov 15, 2023 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hello Everyone,
>
> I would like to start discussion about potential suspension and eventually
> removal of Daskexecutor provider.
>
> After some recent changes and moving DaskExecutor from core to provider we
> have now an open option to suspend and eventually remove Dask Executor from
> our codebase.
>
> After we have the process tested and tried with Qubole, for me this is the
> next candidate to remove and get rid of some burden it involves.
>
> I believe this executor is hardly used by anyone - IMHO it does not really
> add a value to our Executor's offering and it causes a number of CI/
> maintenance issues for us, while we have little to no experience and
> incentive and knowledge to manage it. It slows us down, makes our CI flaky,
> sometimes delays our release process etc etc.
>
> Some of the past discussions about it (this is the third time I am (re)
> starting this discussion - the first time it was in 2020 when we discussed
> removing it for Airflow 2.0.
>
> * https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m
> * https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh
>
> Some of the recent problems with Dask executor impacting stability of our
> CI / tests/ impacting release process are manifested in those issues and
> PRs over the last few years (and this is just a subset of issues and PRs
> dealing with the stability of it.
>
> * https://github.com/apache/airflow/pull/35659
> * https://github.com/apache/airflow/issues/32778
> * https://github.com/apache/airflow/pull/35473
> * https://github.com/apache/airflow/pull/32991
> * https://github.com/apache/airflow/pull/30258
> * https://github.com/apache/airflow/pull/22046
> * https://github.com/apache/airflow/pull/22017
>
>
> Why now? What changed?
>
> One of the fantastic properties of the "providers" approach and moving the
> executor from core to provider - we can now ACTUALLY remove DaskExecutor
> without breaking any of our SemVer promises. This is something I was
> working on for quite a while and with some of the recent changes - making
> our Executor API stable and part of our public interface, introducing full
> life-cycle process for providers this is now entirely possible to remove
> all the maintenance burden while keeping our SemVer promises - so that we
> can do it without bumping Airflow to version 3.
>
> How does it work ? Simply - provider will remain released, and it will
> remain working in the version it has been released for the last time with
> all the future versions of Airflow, But at the same time we can remove the
> code from main of Airflow, stop running tests, stop caring about the
> provider impacting the constraints and generally speaking.
>
> If we decide to do somethin about, we have three options:
>
> 1) Suspend the daskexecutor provider
> 2) Remove it
> 3) We can also follow Suspend -> wait few months -> remove) pattern
>
> All of them are easy, all of them bring us much welcome relief in our
> release and CI and maintenance burden. All of them are well documented and
> tried and all that we need is just to agree which scenario we want to
> follow.
>
> In case 1) we will stop releasing new versions, but in case any new
> "development" is done on the dask side that will break currently released
> providers, it will be on someone who will take on the task to bring it back
> from suspension - fix all dependency issues, make sure tests are green and
> then it's just a matter of PR to bring it back. This is really a way to say
> "Hey Dask Executor community, if you want to keep any future development,
> it's on you to bring it back from suspension and take the task to solve all
> the difficulties we currently have to leave with in Airflow community".
>
> In case 2) we deliberately decide we do not want to maintain it any more
> and if someone would like to bring it back, they will have to convince the
> community and get the VOTE or LAZY CONSENSUS to bring it back (and then it
> could be even different provider package, not compatible etc.).
>
> WDYT? Are there any DaskExecutor users around or someone you know of who
> would rely on it heavily? Are there any reasons why we should keep it as is
> ? Which of the above scenarios would you like to see?
>
> I would love to hear your opinions on that. If we see that we would like to
> follow 1) or 2) I am happy to pass the message to the Dask community and
> let them know that they will need to be involved in case they will want to
> bring back Dask from suspension or removal - I did that before for some of
> the past issues and I am happy to be the messenger (hopefull no-one shoots
> me).
>
> J.
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

["Ferruzzi, Dennis" <fe...@amazon.com.INVALID>]

Just getting back from some time away, if it's till up for discussion, I'd vote for staged removal, the same as any other provider package being dropped.  Perhaps the intermediate step might shake loose a user who cares enough to take it on.  If not, we lose nothing but a little time.


 - ferruzzi


________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Monday, November 20, 2023 8:47 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend (Remove?) Daskexecutor provider

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.



Oh yes. They (or at least one maintainer and looks like quite a few
reacting people) are empathising with us and have no objections for removal
so far, and seem to be grateful for the time I spent explaining what we are
planning and why. I think it also shows it's been worth doing it, if only
to keep good karma and relationships.

I strongly believe in transparency and being candid about things like that.
We might think that people will object but since we have good reasons for
removal and we spent time on discussing and showing that, and we have stats
backing up our data I think it's no brainer also for their community.

I will wait until week passes and will start removal vote - unless
something new comes up.

J.

On Mon, Nov 20, 2023 at 5:07 PM Kaxil Naik <ka...@gmail.com> wrote:

> Looks like they would vote for removal too
> <https://github.com/dask/community/issues/355#issuecomment-1816805948>
>
> On Fri, 17 Nov 2023 at 17:06, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I was asked to open the issue in GitHub to get more visibility by Dask
> > developers so here it is https://github.com/dask/community/issues/355
> >
> > On Fri, Nov 17, 2023 at 1:21 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > OK. Seeing that - I think I will do the next step - I pointed this
> > > discussion to at the discord of Dask and see if there is a volunteer
> > there
> > > who would like to take on fixing the "Quarantined" issue
> > > https://github.com/apache/airflow/issues/32778 -> we have the flaky
> > tests
> > > currently marked as "Quarantined - so it is not really running in
> regular
> > > tests anyway. If there is no-one stepping in and wanting to commit to
> > take
> > > over and manage fixing of similar issues, I will start a voting thread
> > for
> > > removal,
> > >
> > > For me that is an ultimate (and very concrete) "checkpoint"  if the
> Dask
> > > community is up to the task in our repo. And if they are not - they are
> > > always free to fork it and maintain their own provider. I think it's a
> > very
> > > fair approach and quite reasonable expectation - taking into account
> that
> > > we have so little usage and so many problems.
> > >
> > > I announced our intentions in this thread:
> > >
> >
> https://dask.discourse.group/t/planned-removal-of-the-daskexecutor-provider-in-airflow/2328
> > >
> > > I explained the option Dask Community has and gave them a week for
> > > reaction. If no-one steps up and starts seriously diagnosing and fixing
> > the
> > > flaky issue - I will start the vote on removal in about a week.
> > >
> > > J.
> > >
> > >
> > >
> > > On Fri, Nov 17, 2023 at 10:28 AM Wei Lee <we...@gmail.com> wrote:
> > >
> > >> +1 for removal if there is no active maintainer on this one
> > >>
> > >> Best,
> > >> Wei
> > >>
> > >> > On Nov 17, 2023, at 6:00 PM, Andrey Anshin <
> andrey.anshin@taragol.is>
> > >> wrote:
> > >> >
> > >> > +1 for suspend and after a while remove
> > >> >
> > >> > There is a small chance that things would improve after we suspended
> > >> this
> > >> > provider.
> > >> >
> > >> > However we do not have a lot of statistics, AFAIK (correct me if I'm
> > >> wrong)
> > >> > it is only two cases:
> > >> > * 1 provider suspended in the past and restored (Yandex), there is
> no
> > >> > process of removal at that moment.
> > >> > * 1 provider suspended and removed (Qubole)
> > >> >
> > >> >
> > >> > ----
> > >> > Best Wishes
> > >> > *Andrey Anshin*
> > >> >
> > >> >
> > >> >
> > >> > On Fri, 17 Nov 2023 at 12:17, Amogh Desai <amoghdesai.oss@gmail.com
> >
> > >> wrote:
> > >> >
> > >> >> Theres very little incentive in maintaining this if theres no one
> > >> actively
> > >> >> maintaining it.
> > >> >>
> > >> >> I am totally for the removal +1
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty
> > >> >> <co...@astronomer.io.invalid>
> > >> >> wrote:
> > >> >>
> > >> >>> +1 for removal
> > >> >>>
> > >> >>> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr>
> > >> wrote:
> > >> >>>
> > >> >>>>> we would do it branching off at the TAG when the last release
> > >> >> happened
> > >> >>>> and develop/release a fix from there.
> > >> >>>>
> > >> >>>> Since it would be possible to release security patches, +1 to
> > remove
> > >> >> it.
> > >> >>>>
> > >> >>>> On Thu 16 Nov 2023 at 21:22, Oliveira, Niko
> > >> >> <onikolas@amazon.com.invalid
> > >> >>>>
> > >> >>>> wrote:
> > >> >>>>
> > >> >>>>> If no one comes forward willing to support the executor long
> term
> > >> I'm
> > >> >>> +1
> > >> >>>>> for removal.
> > >> >>>>>
> > >> >>>>> ________________________________
> > >> >>>>> From: Vincent Beck <vi...@apache.org>
> > >> >>>>> Sent: Thursday, November 16, 2023 10:59:40 AM
> > >> >>>>> To: dev@airflow.apache.org
> > >> >>>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend
> > >> >> (Remove?)
> > >> >>>>> Daskexecutor provider
> > >> >>>>>
> > >> >>>>> CAUTION: This email originated from outside of the organization.
> > Do
> > >> >> not
> > >> >>>>> click links or open attachments unless you can confirm the
> sender
> > >> and
> > >> >>>> know
> > >> >>>>> the content is safe.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > >> >>> externe.
> > >> >>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> vous
> > ne
> > >> >>>> pouvez
> > >> >>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > >> >> certain
> > >> >>>> que
> > >> >>>>> le contenu ne présente aucun risque.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> +1 for removal
> > >> >>>>>
> > >> >>>>> On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> > >> >>>>>> More detailed comparison:
> > >> >>>>>>
> > >> >>>>>> apache-airflow 2.7.* ~ 255.000 downloads/day
> > >> >>>>>> apache-aurflow-provider-dask-executor ~ 900/ day
> > >> >>>>>>
> > >> >>>>>> This means that *apache-airflow-providers-daskexecutor * is
> > >> >>> downloaded
> > >> >>>> in
> > >> >>>>>> less of *0.3 %* cases, comparing to *apache-airflow*
> > >> >>>>>>
> > >> >>>>>> I'd say it's negligible usage.
> > >> >>>>>>
> > >> >>>>>> My personal vote would go for immediate removal.
> > >> >>>>>>
> > >> >>>>>> WDYT?
> > >> >>>>>>
> > >> >>>>>> J.
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <
> jarek@potiuk.com>
> > >> >>>> wrote:
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <
> eladkal@gmail.com>
> > >> >>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>> Now that the code is in its own provider we can check the
> > >> >> download
> > >> >>>>> stats
> > >> >>>>>>>> of
> > >> >>>>>>>> the library via Pypi stats.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>> Good point but this is only an indication.
> > >> >>>>>>>
> > >> >>>>>>> Currently this is only for Airflow 2.7+ and it is a bit
> > difficult
> > >> >>> to
> > >> >>>>>>> compare those. The "fixed" numbers are misleading (we still do
> > >> >> not
> > >> >>>> know
> > >> >>>>>>> where the 400.000 downloads/day jump to almost 1Million / day
> > >> >> came
> > >> >>>> from
> > >> >>>>>>> mid-October.
> > >> >>>>>>> The ratio is now 1000/day (daskexecutor): 800.000 day
> (airflow)
> > >> >> ->
> > >> >>>>> ~0.1
> > >> >>>>>>> %. But those are only downloads - and not whether it's used -
> > >> >>> likely
> > >> >>>>> there
> > >> >>>>>>> are a number of cases where CI pulls "all" extras.
> > >> >>>>>>>
> > >> >>>>>>> J.
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>>
> > >> ---------------------------------------------------------------------
> > >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >> >>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>
> > >>
> >
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

Oh yes. They (or at least one maintainer and looks like quite a few
reacting people) are empathising with us and have no objections for removal
so far, and seem to be grateful for the time I spent explaining what we are
planning and why. I think it also shows it's been worth doing it, if only
to keep good karma and relationships.

I strongly believe in transparency and being candid about things like that.
We might think that people will object but since we have good reasons for
removal and we spent time on discussing and showing that, and we have stats
backing up our data I think it's no brainer also for their community.

I will wait until week passes and will start removal vote - unless
something new comes up.

J.

On Mon, Nov 20, 2023 at 5:07 PM Kaxil Naik <ka...@gmail.com> wrote:

> Looks like they would vote for removal too
> <https://github.com/dask/community/issues/355#issuecomment-1816805948>
>
> On Fri, 17 Nov 2023 at 17:06, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I was asked to open the issue in GitHub to get more visibility by Dask
> > developers so here it is https://github.com/dask/community/issues/355
> >
> > On Fri, Nov 17, 2023 at 1:21 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > OK. Seeing that - I think I will do the next step - I pointed this
> > > discussion to at the discord of Dask and see if there is a volunteer
> > there
> > > who would like to take on fixing the "Quarantined" issue
> > > https://github.com/apache/airflow/issues/32778 -> we have the flaky
> > tests
> > > currently marked as "Quarantined - so it is not really running in
> regular
> > > tests anyway. If there is no-one stepping in and wanting to commit to
> > take
> > > over and manage fixing of similar issues, I will start a voting thread
> > for
> > > removal,
> > >
> > > For me that is an ultimate (and very concrete) "checkpoint"  if the
> Dask
> > > community is up to the task in our repo. And if they are not - they are
> > > always free to fork it and maintain their own provider. I think it's a
> > very
> > > fair approach and quite reasonable expectation - taking into account
> that
> > > we have so little usage and so many problems.
> > >
> > > I announced our intentions in this thread:
> > >
> >
> https://dask.discourse.group/t/planned-removal-of-the-daskexecutor-provider-in-airflow/2328
> > >
> > > I explained the option Dask Community has and gave them a week for
> > > reaction. If no-one steps up and starts seriously diagnosing and fixing
> > the
> > > flaky issue - I will start the vote on removal in about a week.
> > >
> > > J.
> > >
> > >
> > >
> > > On Fri, Nov 17, 2023 at 10:28 AM Wei Lee <we...@gmail.com> wrote:
> > >
> > >> +1 for removal if there is no active maintainer on this one
> > >>
> > >> Best,
> > >> Wei
> > >>
> > >> > On Nov 17, 2023, at 6:00 PM, Andrey Anshin <
> andrey.anshin@taragol.is>
> > >> wrote:
> > >> >
> > >> > +1 for suspend and after a while remove
> > >> >
> > >> > There is a small chance that things would improve after we suspended
> > >> this
> > >> > provider.
> > >> >
> > >> > However we do not have a lot of statistics, AFAIK (correct me if I'm
> > >> wrong)
> > >> > it is only two cases:
> > >> > * 1 provider suspended in the past and restored (Yandex), there is
> no
> > >> > process of removal at that moment.
> > >> > * 1 provider suspended and removed (Qubole)
> > >> >
> > >> >
> > >> > ----
> > >> > Best Wishes
> > >> > *Andrey Anshin*
> > >> >
> > >> >
> > >> >
> > >> > On Fri, 17 Nov 2023 at 12:17, Amogh Desai <amoghdesai.oss@gmail.com
> >
> > >> wrote:
> > >> >
> > >> >> Theres very little incentive in maintaining this if theres no one
> > >> actively
> > >> >> maintaining it.
> > >> >>
> > >> >> I am totally for the removal +1
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty
> > >> >> <co...@astronomer.io.invalid>
> > >> >> wrote:
> > >> >>
> > >> >>> +1 for removal
> > >> >>>
> > >> >>> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr>
> > >> wrote:
> > >> >>>
> > >> >>>>> we would do it branching off at the TAG when the last release
> > >> >> happened
> > >> >>>> and develop/release a fix from there.
> > >> >>>>
> > >> >>>> Since it would be possible to release security patches, +1 to
> > remove
> > >> >> it.
> > >> >>>>
> > >> >>>> On Thu 16 Nov 2023 at 21:22, Oliveira, Niko
> > >> >> <onikolas@amazon.com.invalid
> > >> >>>>
> > >> >>>> wrote:
> > >> >>>>
> > >> >>>>> If no one comes forward willing to support the executor long
> term
> > >> I'm
> > >> >>> +1
> > >> >>>>> for removal.
> > >> >>>>>
> > >> >>>>> ________________________________
> > >> >>>>> From: Vincent Beck <vi...@apache.org>
> > >> >>>>> Sent: Thursday, November 16, 2023 10:59:40 AM
> > >> >>>>> To: dev@airflow.apache.org
> > >> >>>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend
> > >> >> (Remove?)
> > >> >>>>> Daskexecutor provider
> > >> >>>>>
> > >> >>>>> CAUTION: This email originated from outside of the organization.
> > Do
> > >> >> not
> > >> >>>>> click links or open attachments unless you can confirm the
> sender
> > >> and
> > >> >>>> know
> > >> >>>>> the content is safe.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > >> >>> externe.
> > >> >>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> vous
> > ne
> > >> >>>> pouvez
> > >> >>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > >> >> certain
> > >> >>>> que
> > >> >>>>> le contenu ne présente aucun risque.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> +1 for removal
> > >> >>>>>
> > >> >>>>> On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> > >> >>>>>> More detailed comparison:
> > >> >>>>>>
> > >> >>>>>> apache-airflow 2.7.* ~ 255.000 downloads/day
> > >> >>>>>> apache-aurflow-provider-dask-executor ~ 900/ day
> > >> >>>>>>
> > >> >>>>>> This means that *apache-airflow-providers-daskexecutor * is
> > >> >>> downloaded
> > >> >>>> in
> > >> >>>>>> less of *0.3 %* cases, comparing to *apache-airflow*
> > >> >>>>>>
> > >> >>>>>> I'd say it's negligible usage.
> > >> >>>>>>
> > >> >>>>>> My personal vote would go for immediate removal.
> > >> >>>>>>
> > >> >>>>>> WDYT?
> > >> >>>>>>
> > >> >>>>>> J.
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <
> jarek@potiuk.com>
> > >> >>>> wrote:
> > >> >>>>>>
> > >> >>>>>>>
> > >> >>>>>>> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <
> eladkal@gmail.com>
> > >> >>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>> Now that the code is in its own provider we can check the
> > >> >> download
> > >> >>>>> stats
> > >> >>>>>>>> of
> > >> >>>>>>>> the library via Pypi stats.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>> Good point but this is only an indication.
> > >> >>>>>>>
> > >> >>>>>>> Currently this is only for Airflow 2.7+ and it is a bit
> > difficult
> > >> >>> to
> > >> >>>>>>> compare those. The "fixed" numbers are misleading (we still do
> > >> >> not
> > >> >>>> know
> > >> >>>>>>> where the 400.000 downloads/day jump to almost 1Million / day
> > >> >> came
> > >> >>>> from
> > >> >>>>>>> mid-October.
> > >> >>>>>>> The ratio is now 1000/day (daskexecutor): 800.000 day
> (airflow)
> > >> >> ->
> > >> >>>>> ~0.1
> > >> >>>>>>> %. But those are only downloads - and not whether it's used -
> > >> >>> likely
> > >> >>>>> there
> > >> >>>>>>> are a number of cases where CI pulls "all" extras.
> > >> >>>>>>>
> > >> >>>>>>> J.
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>>
> > >> ---------------------------------------------------------------------
> > >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> > >> >>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>
> > >>
> >
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Kaxil Naik <ka...@gmail.com>]

Looks like they would vote for removal too
<https://github.com/dask/community/issues/355#issuecomment-1816805948>

On Fri, 17 Nov 2023 at 17:06, Jarek Potiuk <ja...@potiuk.com> wrote:

> I was asked to open the issue in GitHub to get more visibility by Dask
> developers so here it is https://github.com/dask/community/issues/355
>
> On Fri, Nov 17, 2023 at 1:21 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > OK. Seeing that - I think I will do the next step - I pointed this
> > discussion to at the discord of Dask and see if there is a volunteer
> there
> > who would like to take on fixing the "Quarantined" issue
> > https://github.com/apache/airflow/issues/32778 -> we have the flaky
> tests
> > currently marked as "Quarantined - so it is not really running in regular
> > tests anyway. If there is no-one stepping in and wanting to commit to
> take
> > over and manage fixing of similar issues, I will start a voting thread
> for
> > removal,
> >
> > For me that is an ultimate (and very concrete) "checkpoint"  if the Dask
> > community is up to the task in our repo. And if they are not - they are
> > always free to fork it and maintain their own provider. I think it's a
> very
> > fair approach and quite reasonable expectation - taking into account that
> > we have so little usage and so many problems.
> >
> > I announced our intentions in this thread:
> >
> https://dask.discourse.group/t/planned-removal-of-the-daskexecutor-provider-in-airflow/2328
> >
> > I explained the option Dask Community has and gave them a week for
> > reaction. If no-one steps up and starts seriously diagnosing and fixing
> the
> > flaky issue - I will start the vote on removal in about a week.
> >
> > J.
> >
> >
> >
> > On Fri, Nov 17, 2023 at 10:28 AM Wei Lee <we...@gmail.com> wrote:
> >
> >> +1 for removal if there is no active maintainer on this one
> >>
> >> Best,
> >> Wei
> >>
> >> > On Nov 17, 2023, at 6:00 PM, Andrey Anshin <an...@taragol.is>
> >> wrote:
> >> >
> >> > +1 for suspend and after a while remove
> >> >
> >> > There is a small chance that things would improve after we suspended
> >> this
> >> > provider.
> >> >
> >> > However we do not have a lot of statistics, AFAIK (correct me if I'm
> >> wrong)
> >> > it is only two cases:
> >> > * 1 provider suspended in the past and restored (Yandex), there is no
> >> > process of removal at that moment.
> >> > * 1 provider suspended and removed (Qubole)
> >> >
> >> >
> >> > ----
> >> > Best Wishes
> >> > *Andrey Anshin*
> >> >
> >> >
> >> >
> >> > On Fri, 17 Nov 2023 at 12:17, Amogh Desai <am...@gmail.com>
> >> wrote:
> >> >
> >> >> Theres very little incentive in maintaining this if theres no one
> >> actively
> >> >> maintaining it.
> >> >>
> >> >> I am totally for the removal +1
> >> >>
> >> >>
> >> >>
> >> >> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty
> >> >> <co...@astronomer.io.invalid>
> >> >> wrote:
> >> >>
> >> >>> +1 for removal
> >> >>>
> >> >>> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr>
> >> wrote:
> >> >>>
> >> >>>>> we would do it branching off at the TAG when the last release
> >> >> happened
> >> >>>> and develop/release a fix from there.
> >> >>>>
> >> >>>> Since it would be possible to release security patches, +1 to
> remove
> >> >> it.
> >> >>>>
> >> >>>> On Thu 16 Nov 2023 at 21:22, Oliveira, Niko
> >> >> <onikolas@amazon.com.invalid
> >> >>>>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> If no one comes forward willing to support the executor long term
> >> I'm
> >> >>> +1
> >> >>>>> for removal.
> >> >>>>>
> >> >>>>> ________________________________
> >> >>>>> From: Vincent Beck <vi...@apache.org>
> >> >>>>> Sent: Thursday, November 16, 2023 10:59:40 AM
> >> >>>>> To: dev@airflow.apache.org
> >> >>>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend
> >> >> (Remove?)
> >> >>>>> Daskexecutor provider
> >> >>>>>
> >> >>>>> CAUTION: This email originated from outside of the organization.
> Do
> >> >> not
> >> >>>>> click links or open attachments unless you can confirm the sender
> >> and
> >> >>>> know
> >> >>>>> the content is safe.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> >> >>> externe.
> >> >>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne
> >> >>>> pouvez
> >> >>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> >> >> certain
> >> >>>> que
> >> >>>>> le contenu ne présente aucun risque.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> +1 for removal
> >> >>>>>
> >> >>>>> On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> >> >>>>>> More detailed comparison:
> >> >>>>>>
> >> >>>>>> apache-airflow 2.7.* ~ 255.000 downloads/day
> >> >>>>>> apache-aurflow-provider-dask-executor ~ 900/ day
> >> >>>>>>
> >> >>>>>> This means that *apache-airflow-providers-daskexecutor * is
> >> >>> downloaded
> >> >>>> in
> >> >>>>>> less of *0.3 %* cases, comparing to *apache-airflow*
> >> >>>>>>
> >> >>>>>> I'd say it's negligible usage.
> >> >>>>>>
> >> >>>>>> My personal vote would go for immediate removal.
> >> >>>>>>
> >> >>>>>> WDYT?
> >> >>>>>>
> >> >>>>>> J.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com>
> >> >>>> wrote:
> >> >>>>>>
> >> >>>>>>>
> >> >>>>>>> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com>
> >> >>>> wrote:
> >> >>>>>>>
> >> >>>>>>>> Now that the code is in its own provider we can check the
> >> >> download
> >> >>>>> stats
> >> >>>>>>>> of
> >> >>>>>>>> the library via Pypi stats.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>> Good point but this is only an indication.
> >> >>>>>>>
> >> >>>>>>> Currently this is only for Airflow 2.7+ and it is a bit
> difficult
> >> >>> to
> >> >>>>>>> compare those. The "fixed" numbers are misleading (we still do
> >> >> not
> >> >>>> know
> >> >>>>>>> where the 400.000 downloads/day jump to almost 1Million / day
> >> >> came
> >> >>>> from
> >> >>>>>>> mid-October.
> >> >>>>>>> The ratio is now 1000/day (daskexecutor): 800.000 day (airflow)
> >> >> ->
> >> >>>>> ~0.1
> >> >>>>>>> %. But those are only downloads - and not whether it's used -
> >> >>> likely
> >> >>>>> there
> >> >>>>>>> are a number of cases where CI pulls "all" extras.
> >> >>>>>>>
> >> >>>>>>> J.
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>>
> >> ---------------------------------------------------------------------
> >> >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >> >>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> For additional commands, e-mail: dev-help@airflow.apache.org
> >>
> >>
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

I was asked to open the issue in GitHub to get more visibility by Dask
developers so here it is https://github.com/dask/community/issues/355

On Fri, Nov 17, 2023 at 1:21 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> OK. Seeing that - I think I will do the next step - I pointed this
> discussion to at the discord of Dask and see if there is a volunteer there
> who would like to take on fixing the "Quarantined" issue
> https://github.com/apache/airflow/issues/32778 -> we have the flaky tests
> currently marked as "Quarantined - so it is not really running in regular
> tests anyway. If there is no-one stepping in and wanting to commit to take
> over and manage fixing of similar issues, I will start a voting thread for
> removal,
>
> For me that is an ultimate (and very concrete) "checkpoint"  if the Dask
> community is up to the task in our repo. And if they are not - they are
> always free to fork it and maintain their own provider. I think it's a very
> fair approach and quite reasonable expectation - taking into account that
> we have so little usage and so many problems.
>
> I announced our intentions in this thread:
> https://dask.discourse.group/t/planned-removal-of-the-daskexecutor-provider-in-airflow/2328
>
> I explained the option Dask Community has and gave them a week for
> reaction. If no-one steps up and starts seriously diagnosing and fixing the
> flaky issue - I will start the vote on removal in about a week.
>
> J.
>
>
>
> On Fri, Nov 17, 2023 at 10:28 AM Wei Lee <we...@gmail.com> wrote:
>
>> +1 for removal if there is no active maintainer on this one
>>
>> Best,
>> Wei
>>
>> > On Nov 17, 2023, at 6:00 PM, Andrey Anshin <an...@taragol.is>
>> wrote:
>> >
>> > +1 for suspend and after a while remove
>> >
>> > There is a small chance that things would improve after we suspended
>> this
>> > provider.
>> >
>> > However we do not have a lot of statistics, AFAIK (correct me if I'm
>> wrong)
>> > it is only two cases:
>> > * 1 provider suspended in the past and restored (Yandex), there is no
>> > process of removal at that moment.
>> > * 1 provider suspended and removed (Qubole)
>> >
>> >
>> > ----
>> > Best Wishes
>> > *Andrey Anshin*
>> >
>> >
>> >
>> > On Fri, 17 Nov 2023 at 12:17, Amogh Desai <am...@gmail.com>
>> wrote:
>> >
>> >> Theres very little incentive in maintaining this if theres no one
>> actively
>> >> maintaining it.
>> >>
>> >> I am totally for the removal +1
>> >>
>> >>
>> >>
>> >> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty
>> >> <co...@astronomer.io.invalid>
>> >> wrote:
>> >>
>> >>> +1 for removal
>> >>>
>> >>> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr>
>> wrote:
>> >>>
>> >>>>> we would do it branching off at the TAG when the last release
>> >> happened
>> >>>> and develop/release a fix from there.
>> >>>>
>> >>>> Since it would be possible to release security patches, +1 to remove
>> >> it.
>> >>>>
>> >>>> On Thu 16 Nov 2023 at 21:22, Oliveira, Niko
>> >> <onikolas@amazon.com.invalid
>> >>>>
>> >>>> wrote:
>> >>>>
>> >>>>> If no one comes forward willing to support the executor long term
>> I'm
>> >>> +1
>> >>>>> for removal.
>> >>>>>
>> >>>>> ________________________________
>> >>>>> From: Vincent Beck <vi...@apache.org>
>> >>>>> Sent: Thursday, November 16, 2023 10:59:40 AM
>> >>>>> To: dev@airflow.apache.org
>> >>>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend
>> >> (Remove?)
>> >>>>> Daskexecutor provider
>> >>>>>
>> >>>>> CAUTION: This email originated from outside of the organization. Do
>> >> not
>> >>>>> click links or open attachments unless you can confirm the sender
>> and
>> >>>> know
>> >>>>> the content is safe.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>> >>> externe.
>> >>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
>> >>>> pouvez
>> >>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>> >> certain
>> >>>> que
>> >>>>> le contenu ne présente aucun risque.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> +1 for removal
>> >>>>>
>> >>>>> On 2023/11/16 18:54:15 Jarek Potiuk wrote:
>> >>>>>> More detailed comparison:
>> >>>>>>
>> >>>>>> apache-airflow 2.7.* ~ 255.000 downloads/day
>> >>>>>> apache-aurflow-provider-dask-executor ~ 900/ day
>> >>>>>>
>> >>>>>> This means that *apache-airflow-providers-daskexecutor * is
>> >>> downloaded
>> >>>> in
>> >>>>>> less of *0.3 %* cases, comparing to *apache-airflow*
>> >>>>>>
>> >>>>>> I'd say it's negligible usage.
>> >>>>>>
>> >>>>>> My personal vote would go for immediate removal.
>> >>>>>>
>> >>>>>> WDYT?
>> >>>>>>
>> >>>>>> J.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com>
>> >>>> wrote:
>> >>>>>>
>> >>>>>>>
>> >>>>>>> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com>
>> >>>> wrote:
>> >>>>>>>
>> >>>>>>>> Now that the code is in its own provider we can check the
>> >> download
>> >>>>> stats
>> >>>>>>>> of
>> >>>>>>>> the library via Pypi stats.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>> Good point but this is only an indication.
>> >>>>>>>
>> >>>>>>> Currently this is only for Airflow 2.7+ and it is a bit difficult
>> >>> to
>> >>>>>>> compare those. The "fixed" numbers are misleading (we still do
>> >> not
>> >>>> know
>> >>>>>>> where the 400.000 downloads/day jump to almost 1Million / day
>> >> came
>> >>>> from
>> >>>>>>> mid-October.
>> >>>>>>> The ratio is now 1000/day (daskexecutor): 800.000 day (airflow)
>> >> ->
>> >>>>> ~0.1
>> >>>>>>> %. But those are only downloads - and not whether it's used -
>> >>> likely
>> >>>>> there
>> >>>>>>> are a number of cases where CI pulls "all" extras.
>> >>>>>>>
>> >>>>>>> J.
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

OK. Seeing that - I think I will do the next step - I pointed this
discussion to at the discord of Dask and see if there is a volunteer there
who would like to take on fixing the "Quarantined" issue
https://github.com/apache/airflow/issues/32778 -> we have the flaky tests
currently marked as "Quarantined - so it is not really running in regular
tests anyway. If there is no-one stepping in and wanting to commit to take
over and manage fixing of similar issues, I will start a voting thread for
removal,

For me that is an ultimate (and very concrete) "checkpoint"  if the Dask
community is up to the task in our repo. And if they are not - they are
always free to fork it and maintain their own provider. I think it's a very
fair approach and quite reasonable expectation - taking into account that
we have so little usage and so many problems.

I announced our intentions in this thread:
https://dask.discourse.group/t/planned-removal-of-the-daskexecutor-provider-in-airflow/2328

I explained the option Dask Community has and gave them a week for
reaction. If no-one steps up and starts seriously diagnosing and fixing the
flaky issue - I will start the vote on removal in about a week.

J.



On Fri, Nov 17, 2023 at 10:28 AM Wei Lee <we...@gmail.com> wrote:

> +1 for removal if there is no active maintainer on this one
>
> Best,
> Wei
>
> > On Nov 17, 2023, at 6:00 PM, Andrey Anshin <an...@taragol.is>
> wrote:
> >
> > +1 for suspend and after a while remove
> >
> > There is a small chance that things would improve after we suspended this
> > provider.
> >
> > However we do not have a lot of statistics, AFAIK (correct me if I'm
> wrong)
> > it is only two cases:
> > * 1 provider suspended in the past and restored (Yandex), there is no
> > process of removal at that moment.
> > * 1 provider suspended and removed (Qubole)
> >
> >
> > ----
> > Best Wishes
> > *Andrey Anshin*
> >
> >
> >
> > On Fri, 17 Nov 2023 at 12:17, Amogh Desai <am...@gmail.com>
> wrote:
> >
> >> Theres very little incentive in maintaining this if theres no one
> actively
> >> maintaining it.
> >>
> >> I am totally for the removal +1
> >>
> >>
> >>
> >> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty
> >> <co...@astronomer.io.invalid>
> >> wrote:
> >>
> >>> +1 for removal
> >>>
> >>> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr>
> wrote:
> >>>
> >>>>> we would do it branching off at the TAG when the last release
> >> happened
> >>>> and develop/release a fix from there.
> >>>>
> >>>> Since it would be possible to release security patches, +1 to remove
> >> it.
> >>>>
> >>>> On Thu 16 Nov 2023 at 21:22, Oliveira, Niko
> >> <onikolas@amazon.com.invalid
> >>>>
> >>>> wrote:
> >>>>
> >>>>> If no one comes forward willing to support the executor long term I'm
> >>> +1
> >>>>> for removal.
> >>>>>
> >>>>> ________________________________
> >>>>> From: Vincent Beck <vi...@apache.org>
> >>>>> Sent: Thursday, November 16, 2023 10:59:40 AM
> >>>>> To: dev@airflow.apache.org
> >>>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend
> >> (Remove?)
> >>>>> Daskexecutor provider
> >>>>>
> >>>>> CAUTION: This email originated from outside of the organization. Do
> >> not
> >>>>> click links or open attachments unless you can confirm the sender and
> >>>> know
> >>>>> the content is safe.
> >>>>>
> >>>>>
> >>>>>
> >>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> >>> externe.
> >>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> >>>> pouvez
> >>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> >> certain
> >>>> que
> >>>>> le contenu ne présente aucun risque.
> >>>>>
> >>>>>
> >>>>>
> >>>>> +1 for removal
> >>>>>
> >>>>> On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> >>>>>> More detailed comparison:
> >>>>>>
> >>>>>> apache-airflow 2.7.* ~ 255.000 downloads/day
> >>>>>> apache-aurflow-provider-dask-executor ~ 900/ day
> >>>>>>
> >>>>>> This means that *apache-airflow-providers-daskexecutor * is
> >>> downloaded
> >>>> in
> >>>>>> less of *0.3 %* cases, comparing to *apache-airflow*
> >>>>>>
> >>>>>> I'd say it's negligible usage.
> >>>>>>
> >>>>>> My personal vote would go for immediate removal.
> >>>>>>
> >>>>>> WDYT?
> >>>>>>
> >>>>>> J.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com>
> >>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Now that the code is in its own provider we can check the
> >> download
> >>>>> stats
> >>>>>>>> of
> >>>>>>>> the library via Pypi stats.
> >>>>>>>>
> >>>>>>>>
> >>>>>>> Good point but this is only an indication.
> >>>>>>>
> >>>>>>> Currently this is only for Airflow 2.7+ and it is a bit difficult
> >>> to
> >>>>>>> compare those. The "fixed" numbers are misleading (we still do
> >> not
> >>>> know
> >>>>>>> where the 400.000 downloads/day jump to almost 1Million / day
> >> came
> >>>> from
> >>>>>>> mid-October.
> >>>>>>> The ratio is now 1000/day (daskexecutor): 800.000 day (airflow)
> >> ->
> >>>>> ~0.1
> >>>>>>> %. But those are only downloads - and not whether it's used -
> >>> likely
> >>>>> there
> >>>>>>> are a number of cases where CI pulls "all" extras.
> >>>>>>>
> >>>>>>> J.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >>>>> For additional commands, e-mail: dev-help@airflow.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Wei Lee <we...@gmail.com>]

+1 for removal if there is no active maintainer on this one

Best,
Wei

> On Nov 17, 2023, at 6:00 PM, Andrey Anshin <an...@taragol.is> wrote:
> 
> +1 for suspend and after a while remove
> 
> There is a small chance that things would improve after we suspended this
> provider.
> 
> However we do not have a lot of statistics, AFAIK (correct me if I'm wrong)
> it is only two cases:
> * 1 provider suspended in the past and restored (Yandex), there is no
> process of removal at that moment.
> * 1 provider suspended and removed (Qubole)
> 
> 
> ----
> Best Wishes
> *Andrey Anshin*
> 
> 
> 
> On Fri, 17 Nov 2023 at 12:17, Amogh Desai <am...@gmail.com> wrote:
> 
>> Theres very little incentive in maintaining this if theres no one actively
>> maintaining it.
>> 
>> I am totally for the removal +1
>> 
>> 
>> 
>> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty
>> <co...@astronomer.io.invalid>
>> wrote:
>> 
>>> +1 for removal
>>> 
>>> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr> wrote:
>>> 
>>>>> we would do it branching off at the TAG when the last release
>> happened
>>>> and develop/release a fix from there.
>>>> 
>>>> Since it would be possible to release security patches, +1 to remove
>> it.
>>>> 
>>>> On Thu 16 Nov 2023 at 21:22, Oliveira, Niko
>> <onikolas@amazon.com.invalid
>>>> 
>>>> wrote:
>>>> 
>>>>> If no one comes forward willing to support the executor long term I'm
>>> +1
>>>>> for removal.
>>>>> 
>>>>> ________________________________
>>>>> From: Vincent Beck <vi...@apache.org>
>>>>> Sent: Thursday, November 16, 2023 10:59:40 AM
>>>>> To: dev@airflow.apache.org
>>>>> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend
>> (Remove?)
>>>>> Daskexecutor provider
>>>>> 
>>>>> CAUTION: This email originated from outside of the organization. Do
>> not
>>>>> click links or open attachments unless you can confirm the sender and
>>>> know
>>>>> the content is safe.
>>>>> 
>>>>> 
>>>>> 
>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
>>> externe.
>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
>>>> pouvez
>>>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
>> certain
>>>> que
>>>>> le contenu ne présente aucun risque.
>>>>> 
>>>>> 
>>>>> 
>>>>> +1 for removal
>>>>> 
>>>>> On 2023/11/16 18:54:15 Jarek Potiuk wrote:
>>>>>> More detailed comparison:
>>>>>> 
>>>>>> apache-airflow 2.7.* ~ 255.000 downloads/day
>>>>>> apache-aurflow-provider-dask-executor ~ 900/ day
>>>>>> 
>>>>>> This means that *apache-airflow-providers-daskexecutor * is
>>> downloaded
>>>> in
>>>>>> less of *0.3 %* cases, comparing to *apache-airflow*
>>>>>> 
>>>>>> I'd say it's negligible usage.
>>>>>> 
>>>>>> My personal vote would go for immediate removal.
>>>>>> 
>>>>>> WDYT?
>>>>>> 
>>>>>> J.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com>
>>>> wrote:
>>>>>>> 
>>>>>>>> Now that the code is in its own provider we can check the
>> download
>>>>> stats
>>>>>>>> of
>>>>>>>> the library via Pypi stats.
>>>>>>>> 
>>>>>>>> 
>>>>>>> Good point but this is only an indication.
>>>>>>> 
>>>>>>> Currently this is only for Airflow 2.7+ and it is a bit difficult
>>> to
>>>>>>> compare those. The "fixed" numbers are misleading (we still do
>> not
>>>> know
>>>>>>> where the 400.000 downloads/day jump to almost 1Million / day
>> came
>>>> from
>>>>>>> mid-October.
>>>>>>> The ratio is now 1000/day (daskexecutor): 800.000 day (airflow)
>> ->
>>>>> ~0.1
>>>>>>> %. But those are only downloads - and not whether it's used -
>>> likely
>>>>> there
>>>>>>> are a number of cases where CI pulls "all" extras.
>>>>>>> 
>>>>>>> J.
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>> 
>>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Andrey Anshin <an...@taragol.is>]

+1 for suspend and after a while remove

There is a small chance that things would improve after we suspended this
provider.

However we do not have a lot of statistics, AFAIK (correct me if I'm wrong)
it is only two cases:
* 1 provider suspended in the past and restored (Yandex), there is no
process of removal at that moment.
* 1 provider suspended and removed (Qubole)


----
Best Wishes
*Andrey Anshin*



On Fri, 17 Nov 2023 at 12:17, Amogh Desai <am...@gmail.com> wrote:

> Theres very little incentive in maintaining this if theres no one actively
> maintaining it.
>
> I am totally for the removal +1
>
>
>
> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty
> <co...@astronomer.io.invalid>
> wrote:
>
> > +1 for removal
> >
> > On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr> wrote:
> >
> > > > we would do it branching off at the TAG when the last release
> happened
> > > and develop/release a fix from there.
> > >
> > > Since it would be possible to release security patches, +1 to remove
> it.
> > >
> > > On Thu 16 Nov 2023 at 21:22, Oliveira, Niko
> <onikolas@amazon.com.invalid
> > >
> > > wrote:
> > >
> > > > If no one comes forward willing to support the executor long term I'm
> > +1
> > > > for removal.
> > > >
> > > > ________________________________
> > > > From: Vincent Beck <vi...@apache.org>
> > > > Sent: Thursday, November 16, 2023 10:59:40 AM
> > > > To: dev@airflow.apache.org
> > > > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend
> (Remove?)
> > > > Daskexecutor provider
> > > >
> > > > CAUTION: This email originated from outside of the organization. Do
> not
> > > > click links or open attachments unless you can confirm the sender and
> > > know
> > > > the content is safe.
> > > >
> > > >
> > > >
> > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > externe.
> > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> > > pouvez
> > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain
> > > que
> > > > le contenu ne présente aucun risque.
> > > >
> > > >
> > > >
> > > > +1 for removal
> > > >
> > > > On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> > > > > More detailed comparison:
> > > > >
> > > > > apache-airflow 2.7.* ~ 255.000 downloads/day
> > > > > apache-aurflow-provider-dask-executor ~ 900/ day
> > > > >
> > > > > This means that *apache-airflow-providers-daskexecutor * is
> > downloaded
> > > in
> > > > > less of *0.3 %* cases, comparing to *apache-airflow*
> > > > >
> > > > > I'd say it's negligible usage.
> > > > >
> > > > > My personal vote would go for immediate removal.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > >
> > > > > >
> > > > > > On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com>
> > > wrote:
> > > > > >
> > > > > >> Now that the code is in its own provider we can check the
> download
> > > > stats
> > > > > >> of
> > > > > >> the library via Pypi stats.
> > > > > >>
> > > > > >>
> > > > > > Good point but this is only an indication.
> > > > > >
> > > > > > Currently this is only for Airflow 2.7+ and it is a bit difficult
> > to
> > > > > > compare those. The "fixed" numbers are misleading (we still do
> not
> > > know
> > > > > > where the 400.000 downloads/day jump to almost 1Million / day
> came
> > > from
> > > > > > mid-October.
> > > > > > The ratio is now 1000/day (daskexecutor): 800.000 day (airflow)
> ->
> > > > ~0.1
> > > > > > %. But those are only downloads - and not whether it's used -
> > likely
> > > > there
> > > > > > are a number of cases where CI pulls "all" extras.
> > > > > >
> > > > > > J.
> > > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Aritra Basu <ar...@gmail.com>]

+1 All for removing it if it's not in use and is giving us trouble with
maintaining.

--
Regards,
Aritra Basu

On Fri, Nov 17, 2023, 1:47 PM Amogh Desai <am...@gmail.com> wrote:

> Theres very little incentive in maintaining this if theres no one actively
> maintaining it.
>
> I am totally for the removal +1
>
>
>
> On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty
> <co...@astronomer.io.invalid>
> wrote:
>
> > +1 for removal
> >
> > On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr> wrote:
> >
> > > > we would do it branching off at the TAG when the last release
> happened
> > > and develop/release a fix from there.
> > >
> > > Since it would be possible to release security patches, +1 to remove
> it.
> > >
> > > On Thu 16 Nov 2023 at 21:22, Oliveira, Niko
> <onikolas@amazon.com.invalid
> > >
> > > wrote:
> > >
> > > > If no one comes forward willing to support the executor long term I'm
> > +1
> > > > for removal.
> > > >
> > > > ________________________________
> > > > From: Vincent Beck <vi...@apache.org>
> > > > Sent: Thursday, November 16, 2023 10:59:40 AM
> > > > To: dev@airflow.apache.org
> > > > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend
> (Remove?)
> > > > Daskexecutor provider
> > > >
> > > > CAUTION: This email originated from outside of the organization. Do
> not
> > > > click links or open attachments unless you can confirm the sender and
> > > know
> > > > the content is safe.
> > > >
> > > >
> > > >
> > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > externe.
> > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> > > pouvez
> > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain
> > > que
> > > > le contenu ne présente aucun risque.
> > > >
> > > >
> > > >
> > > > +1 for removal
> > > >
> > > > On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> > > > > More detailed comparison:
> > > > >
> > > > > apache-airflow 2.7.* ~ 255.000 downloads/day
> > > > > apache-aurflow-provider-dask-executor ~ 900/ day
> > > > >
> > > > > This means that *apache-airflow-providers-daskexecutor * is
> > downloaded
> > > in
> > > > > less of *0.3 %* cases, comparing to *apache-airflow*
> > > > >
> > > > > I'd say it's negligible usage.
> > > > >
> > > > > My personal vote would go for immediate removal.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > >
> > > > > >
> > > > > > On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com>
> > > wrote:
> > > > > >
> > > > > >> Now that the code is in its own provider we can check the
> download
> > > > stats
> > > > > >> of
> > > > > >> the library via Pypi stats.
> > > > > >>
> > > > > >>
> > > > > > Good point but this is only an indication.
> > > > > >
> > > > > > Currently this is only for Airflow 2.7+ and it is a bit difficult
> > to
> > > > > > compare those. The "fixed" numbers are misleading (we still do
> not
> > > know
> > > > > > where the 400.000 downloads/day jump to almost 1Million / day
> came
> > > from
> > > > > > mid-October.
> > > > > > The ratio is now 1000/day (daskexecutor): 800.000 day (airflow)
> ->
> > > > ~0.1
> > > > > > %. But those are only downloads - and not whether it's used -
> > likely
> > > > there
> > > > > > are a number of cases where CI pulls "all" extras.
> > > > > >
> > > > > > J.
> > > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Amogh Desai <am...@gmail.com>]

Theres very little incentive in maintaining this if theres no one actively
maintaining it.

I am totally for the removal +1



On Fri, 17 Nov 2023 at 1:21 AM, Collin McNulty <co...@astronomer.io.invalid>
wrote:

> +1 for removal
>
> On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr> wrote:
>
> > > we would do it branching off at the TAG when the last release happened
> > and develop/release a fix from there.
> >
> > Since it would be possible to release security patches, +1 to remove it.
> >
> > On Thu 16 Nov 2023 at 21:22, Oliveira, Niko <onikolas@amazon.com.invalid
> >
> > wrote:
> >
> > > If no one comes forward willing to support the executor long term I'm
> +1
> > > for removal.
> > >
> > > ________________________________
> > > From: Vincent Beck <vi...@apache.org>
> > > Sent: Thursday, November 16, 2023 10:59:40 AM
> > > To: dev@airflow.apache.org
> > > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend (Remove?)
> > > Daskexecutor provider
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > > click links or open attachments unless you can confirm the sender and
> > know
> > > the content is safe.
> > >
> > >
> > >
> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe.
> > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> > pouvez
> > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> > que
> > > le contenu ne présente aucun risque.
> > >
> > >
> > >
> > > +1 for removal
> > >
> > > On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> > > > More detailed comparison:
> > > >
> > > > apache-airflow 2.7.* ~ 255.000 downloads/day
> > > > apache-aurflow-provider-dask-executor ~ 900/ day
> > > >
> > > > This means that *apache-airflow-providers-daskexecutor * is
> downloaded
> > in
> > > > less of *0.3 %* cases, comparing to *apache-airflow*
> > > >
> > > > I'd say it's negligible usage.
> > > >
> > > > My personal vote would go for immediate removal.
> > > >
> > > > WDYT?
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > > On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >
> > > > >
> > > > > On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com>
> > wrote:
> > > > >
> > > > >> Now that the code is in its own provider we can check the download
> > > stats
> > > > >> of
> > > > >> the library via Pypi stats.
> > > > >>
> > > > >>
> > > > > Good point but this is only an indication.
> > > > >
> > > > > Currently this is only for Airflow 2.7+ and it is a bit difficult
> to
> > > > > compare those. The "fixed" numbers are misleading (we still do not
> > know
> > > > > where the 400.000 downloads/day jump to almost 1Million / day came
> > from
> > > > > mid-October.
> > > > > The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) ->
> > > ~0.1
> > > > > %. But those are only downloads - and not whether it's used -
> likely
> > > there
> > > > > are a number of cases where CI pulls "all" extras.
> > > > >
> > > > > J.
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Collin McNulty <co...@astronomer.io.INVALID>]

+1 for removal

On Thu, Nov 16, 2023 at 1:43 PM Hussein Awala <hu...@awala.fr> wrote:

> > we would do it branching off at the TAG when the last release happened
> and develop/release a fix from there.
>
> Since it would be possible to release security patches, +1 to remove it.
>
> On Thu 16 Nov 2023 at 21:22, Oliveira, Niko <on...@amazon.com.invalid>
> wrote:
>
> > If no one comes forward willing to support the executor long term I'm +1
> > for removal.
> >
> > ________________________________
> > From: Vincent Beck <vi...@apache.org>
> > Sent: Thursday, November 16, 2023 10:59:40 AM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend (Remove?)
> > Daskexecutor provider
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> > +1 for removal
> >
> > On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> > > More detailed comparison:
> > >
> > > apache-airflow 2.7.* ~ 255.000 downloads/day
> > > apache-aurflow-provider-dask-executor ~ 900/ day
> > >
> > > This means that *apache-airflow-providers-daskexecutor * is downloaded
> in
> > > less of *0.3 %* cases, comparing to *apache-airflow*
> > >
> > > I'd say it's negligible usage.
> > >
> > > My personal vote would go for immediate removal.
> > >
> > > WDYT?
> > >
> > > J.
> > >
> > >
> > >
> > > On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >
> > > >
> > > > On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com>
> wrote:
> > > >
> > > >> Now that the code is in its own provider we can check the download
> > stats
> > > >> of
> > > >> the library via Pypi stats.
> > > >>
> > > >>
> > > > Good point but this is only an indication.
> > > >
> > > > Currently this is only for Airflow 2.7+ and it is a bit difficult to
> > > > compare those. The "fixed" numbers are misleading (we still do not
> know
> > > > where the 400.000 downloads/day jump to almost 1Million / day came
> from
> > > > mid-October.
> > > > The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) ->
> > ~0.1
> > > > %. But those are only downloads - and not whether it's used - likely
> > there
> > > > are a number of cases where CI pulls "all" extras.
> > > >
> > > > J.
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Hussein Awala <hu...@awala.fr>]

> we would do it branching off at the TAG when the last release happened
and develop/release a fix from there.

Since it would be possible to release security patches, +1 to remove it.

On Thu 16 Nov 2023 at 21:22, Oliveira, Niko <on...@amazon.com.invalid>
wrote:

> If no one comes forward willing to support the executor long term I'm +1
> for removal.
>
> ________________________________
> From: Vincent Beck <vi...@apache.org>
> Sent: Thursday, November 16, 2023 10:59:40 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend (Remove?)
> Daskexecutor provider
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 for removal
>
> On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> > More detailed comparison:
> >
> > apache-airflow 2.7.* ~ 255.000 downloads/day
> > apache-aurflow-provider-dask-executor ~ 900/ day
> >
> > This means that *apache-airflow-providers-daskexecutor * is downloaded in
> > less of *0.3 %* cases, comparing to *apache-airflow*
> >
> > I'd say it's negligible usage.
> >
> > My personal vote would go for immediate removal.
> >
> > WDYT?
> >
> > J.
> >
> >
> >
> > On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > >
> > > On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com> wrote:
> > >
> > >> Now that the code is in its own provider we can check the download
> stats
> > >> of
> > >> the library via Pypi stats.
> > >>
> > >>
> > > Good point but this is only an indication.
> > >
> > > Currently this is only for Airflow 2.7+ and it is a bit difficult to
> > > compare those. The "fixed" numbers are misleading (we still do not know
> > > where the 400.000 downloads/day jump to almost 1Million / day came from
> > > mid-October.
> > > The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) ->
> ~0.1
> > > %. But those are only downloads - and not whether it's used - likely
> there
> > > are a number of cases where CI pulls "all" extras.
> > >
> > > J.
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

["Oliveira, Niko" <on...@amazon.com.INVALID>]

If no one comes forward willing to support the executor long term I'm +1 for removal.

________________________________
From: Vincent Beck <vi...@apache.org>
Sent: Thursday, November 16, 2023 10:59:40 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Suspend (Remove?) Daskexecutor provider

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.



+1 for removal

On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> More detailed comparison:
>
> apache-airflow 2.7.* ~ 255.000 downloads/day
> apache-aurflow-provider-dask-executor ~ 900/ day
>
> This means that *apache-airflow-providers-daskexecutor * is downloaded in
> less of *0.3 %* cases, comparing to *apache-airflow*
>
> I'd say it's negligible usage.
>
> My personal vote would go for immediate removal.
>
> WDYT?
>
> J.
>
>
>
> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> >
> > On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com> wrote:
> >
> >> Now that the code is in its own provider we can check the download stats
> >> of
> >> the library via Pypi stats.
> >>
> >>
> > Good point but this is only an indication.
> >
> > Currently this is only for Airflow 2.7+ and it is a bit difficult to
> > compare those. The "fixed" numbers are misleading (we still do not know
> > where the 400.000 downloads/day jump to almost 1Million / day came from
> > mid-October.
> > The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) ->  ~0.1
> > %. But those are only downloads - and not whether it's used - likely there
> > are a number of cases where CI pulls "all" extras.
> >
> > J.
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Vincent Beck <vi...@apache.org>]

+1 for removal

On 2023/11/16 18:54:15 Jarek Potiuk wrote:
> More detailed comparison:
> 
> apache-airflow 2.7.* ~ 255.000 downloads/day
> apache-aurflow-provider-dask-executor ~ 900/ day
> 
> This means that *apache-airflow-providers-daskexecutor * is downloaded in
> less of *0.3 %* cases, comparing to *apache-airflow*
> 
> I'd say it's negligible usage.
> 
> My personal vote would go for immediate removal.
> 
> WDYT?
> 
> J.
> 
> 
> 
> On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> 
> >
> > On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com> wrote:
> >
> >> Now that the code is in its own provider we can check the download stats
> >> of
> >> the library via Pypi stats.
> >>
> >>
> > Good point but this is only an indication.
> >
> > Currently this is only for Airflow 2.7+ and it is a bit difficult to
> > compare those. The "fixed" numbers are misleading (we still do not know
> > where the 400.000 downloads/day jump to almost 1Million / day came from
> > mid-October.
> > The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) ->  ~0.1
> > %. But those are only downloads - and not whether it's used - likely there
> > are a number of cases where CI pulls "all" extras.
> >
> > J.
> >
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

More detailed comparison:

apache-airflow 2.7.* ~ 255.000 downloads/day
apache-aurflow-provider-dask-executor ~ 900/ day

This means that *apache-airflow-providers-daskexecutor * is downloaded in
less of *0.3 %* cases, comparing to *apache-airflow*

I'd say it's negligible usage.

My personal vote would go for immediate removal.

WDYT?

J.



On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com> wrote:

>
> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com> wrote:
>
>> Now that the code is in its own provider we can check the download stats
>> of
>> the library via Pypi stats.
>>
>>
> Good point but this is only an indication.
>
> Currently this is only for Airflow 2.7+ and it is a bit difficult to
> compare those. The "fixed" numbers are misleading (we still do not know
> where the 400.000 downloads/day jump to almost 1Million / day came from
> mid-October.
> The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) ->  ~0.1
> %. But those are only downloads - and not whether it's used - likely there
> are a number of cases where CI pulls "all" extras.
>
> J.
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

More detailed comparison:

apache-airflow 2.7.* ~ 255.000 downloads/day
apache-aurflow-provider-dask-executor ~ 900/ day

This means that *apache-airflow-providers-daskexecutor * is downloaded in
less of *0.3 %* cases, comparing to *apache-airflow*

I'd say it's negligible usage.

My personal vote would go for immediate removal.

WDYT?

J.



On Wed, Nov 15, 2023 at 10:39 PM Jarek Potiuk <ja...@potiuk.com> wrote:

>
> On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com> wrote:
>
>> Now that the code is in its own provider we can check the download stats
>> of
>> the library via Pypi stats.
>>
>>
> Good point but this is only an indication.
>
> Currently this is only for Airflow 2.7+ and it is a bit difficult to
> compare those. The "fixed" numbers are misleading (we still do not know
> where the 400.000 downloads/day jump to almost 1Million / day came from
> mid-October.
> The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) ->  ~0.1
> %. But those are only downloads - and not whether it's used - likely there
> are a number of cases where CI pulls "all" extras.
>
> J.
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com> wrote:

> Now that the code is in its own provider we can check the download stats of
> the library via Pypi stats.
>
>
Good point but this is only an indication.

Currently this is only for Airflow 2.7+ and it is a bit difficult to
compare those. The "fixed" numbers are misleading (we still do not know
where the 400.000 downloads/day jump to almost 1Million / day came from
mid-October.
The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) ->  ~0.1 %.
But those are only downloads - and not whether it's used - likely there are
a number of cases where CI pulls "all" extras.

J.

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Jarek Potiuk <ja...@potiuk.com>]

On Wed, Nov 15, 2023 at 10:20 PM Elad Kalif <el...@gmail.com> wrote:

> Now that the code is in its own provider we can check the download stats of
> the library via Pypi stats.
>
>
Good point but this is only an indication.

Currently this is only for Airflow 2.7+ and it is a bit difficult to
compare those. The "fixed" numbers are misleading (we still do not know
where the 400.000 downloads/day jump to almost 1Million / day came from
mid-October.
The ratio is now 1000/day (daskexecutor): 800.000 day (airflow) ->  ~0.1 %.
But those are only downloads - and not whether it's used - likely there are
a number of cases where CI pulls "all" extras.

J.

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Elad Kalif <el...@gmail.com>]

Now that the code is in its own provider we can check the download stats of
the library via Pypi stats.


בתאריך יום ד׳, 15 בנוב׳ 2023, 15:32, מאת Jarek Potiuk ‏<ja...@potiuk.com>:

> Hello Everyone,
>
> I would like to start discussion about potential suspension and eventually
> removal of Daskexecutor provider.
>
> After some recent changes and moving DaskExecutor from core to provider we
> have now an open option to suspend and eventually remove Dask Executor from
> our codebase.
>
> After we have the process tested and tried with Qubole, for me this is the
> next candidate to remove and get rid of some burden it involves.
>
> I believe this executor is hardly used by anyone - IMHO it does not really
> add a value to our Executor's offering and it causes a number of CI/
> maintenance issues for us, while we have little to no experience and
> incentive and knowledge to manage it. It slows us down, makes our CI flaky,
> sometimes delays our release process etc etc.
>
> Some of the past discussions about it (this is the third time I am (re)
> starting this discussion - the first time it was in 2020 when we discussed
> removing it for Airflow 2.0.
>
> * https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m
> * https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh
>
> Some of the recent problems with Dask executor impacting stability of our
> CI / tests/ impacting release process are manifested in those issues and
> PRs over the last few years (and this is just a subset of issues and PRs
> dealing with the stability of it.
>
> * https://github.com/apache/airflow/pull/35659
> * https://github.com/apache/airflow/issues/32778
> * https://github.com/apache/airflow/pull/35473
> * https://github.com/apache/airflow/pull/32991
> * https://github.com/apache/airflow/pull/30258
> * https://github.com/apache/airflow/pull/22046
> * https://github.com/apache/airflow/pull/22017
>
>
> Why now? What changed?
>
> One of the fantastic properties of the "providers" approach and moving the
> executor from core to provider - we can now ACTUALLY remove DaskExecutor
> without breaking any of our SemVer promises. This is something I was
> working on for quite a while and with some of the recent changes - making
> our Executor API stable and part of our public interface, introducing full
> life-cycle process for providers this is now entirely possible to remove
> all the maintenance burden while keeping our SemVer promises - so that we
> can do it without bumping Airflow to version 3.
>
> How does it work ? Simply - provider will remain released, and it will
> remain working in the version it has been released for the last time with
> all the future versions of Airflow, But at the same time we can remove the
> code from main of Airflow, stop running tests, stop caring about the
> provider impacting the constraints and generally speaking.
>
> If we decide to do somethin about, we have three options:
>
> 1) Suspend the daskexecutor provider
> 2) Remove it
> 3) We can also follow Suspend -> wait few months -> remove) pattern
>
> All of them are easy, all of them bring us much welcome relief in our
> release and CI and maintenance burden. All of them are well documented and
> tried and all that we need is just to agree which scenario we want to
> follow.
>
> In case 1) we will stop releasing new versions, but in case any new
> "development" is done on the dask side that will break currently released
> providers, it will be on someone who will take on the task to bring it back
> from suspension - fix all dependency issues, make sure tests are green and
> then it's just a matter of PR to bring it back. This is really a way to say
> "Hey Dask Executor community, if you want to keep any future development,
> it's on you to bring it back from suspension and take the task to solve all
> the difficulties we currently have to leave with in Airflow community".
>
> In case 2) we deliberately decide we do not want to maintain it any more
> and if someone would like to bring it back, they will have to convince the
> community and get the VOTE or LAZY CONSENSUS to bring it back (and then it
> could be even different provider package, not compatible etc.).
>
> WDYT? Are there any DaskExecutor users around or someone you know of who
> would rely on it heavily? Are there any reasons why we should keep it as is
> ? Which of the above scenarios would you like to see?
>
> I would love to hear your opinions on that. If we see that we would like to
> follow 1) or 2) I am happy to pass the message to the Dask community and
> let them know that they will need to be involved in case they will want to
> bring back Dask from suspension or removal - I did that before for some of
> the past issues and I am happy to be the messenger (hopefull no-one shoots
> me).
>
> J.
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Elad Kalif <el...@gmail.com>]

Now that the code is in its own provider we can check the download stats of
the library via Pypi stats.


בתאריך יום ד׳, 15 בנוב׳ 2023, 15:32, מאת Jarek Potiuk ‏<ja...@potiuk.com>:

> Hello Everyone,
>
> I would like to start discussion about potential suspension and eventually
> removal of Daskexecutor provider.
>
> After some recent changes and moving DaskExecutor from core to provider we
> have now an open option to suspend and eventually remove Dask Executor from
> our codebase.
>
> After we have the process tested and tried with Qubole, for me this is the
> next candidate to remove and get rid of some burden it involves.
>
> I believe this executor is hardly used by anyone - IMHO it does not really
> add a value to our Executor's offering and it causes a number of CI/
> maintenance issues for us, while we have little to no experience and
> incentive and knowledge to manage it. It slows us down, makes our CI flaky,
> sometimes delays our release process etc etc.
>
> Some of the past discussions about it (this is the third time I am (re)
> starting this discussion - the first time it was in 2020 when we discussed
> removing it for Airflow 2.0.
>
> * https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m
> * https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh
>
> Some of the recent problems with Dask executor impacting stability of our
> CI / tests/ impacting release process are manifested in those issues and
> PRs over the last few years (and this is just a subset of issues and PRs
> dealing with the stability of it.
>
> * https://github.com/apache/airflow/pull/35659
> * https://github.com/apache/airflow/issues/32778
> * https://github.com/apache/airflow/pull/35473
> * https://github.com/apache/airflow/pull/32991
> * https://github.com/apache/airflow/pull/30258
> * https://github.com/apache/airflow/pull/22046
> * https://github.com/apache/airflow/pull/22017
>
>
> Why now? What changed?
>
> One of the fantastic properties of the "providers" approach and moving the
> executor from core to provider - we can now ACTUALLY remove DaskExecutor
> without breaking any of our SemVer promises. This is something I was
> working on for quite a while and with some of the recent changes - making
> our Executor API stable and part of our public interface, introducing full
> life-cycle process for providers this is now entirely possible to remove
> all the maintenance burden while keeping our SemVer promises - so that we
> can do it without bumping Airflow to version 3.
>
> How does it work ? Simply - provider will remain released, and it will
> remain working in the version it has been released for the last time with
> all the future versions of Airflow, But at the same time we can remove the
> code from main of Airflow, stop running tests, stop caring about the
> provider impacting the constraints and generally speaking.
>
> If we decide to do somethin about, we have three options:
>
> 1) Suspend the daskexecutor provider
> 2) Remove it
> 3) We can also follow Suspend -> wait few months -> remove) pattern
>
> All of them are easy, all of them bring us much welcome relief in our
> release and CI and maintenance burden. All of them are well documented and
> tried and all that we need is just to agree which scenario we want to
> follow.
>
> In case 1) we will stop releasing new versions, but in case any new
> "development" is done on the dask side that will break currently released
> providers, it will be on someone who will take on the task to bring it back
> from suspension - fix all dependency issues, make sure tests are green and
> then it's just a matter of PR to bring it back. This is really a way to say
> "Hey Dask Executor community, if you want to keep any future development,
> it's on you to bring it back from suspension and take the task to solve all
> the difficulties we currently have to leave with in Airflow community".
>
> In case 2) we deliberately decide we do not want to maintain it any more
> and if someone would like to bring it back, they will have to convince the
> community and get the VOTE or LAZY CONSENSUS to bring it back (and then it
> could be even different provider package, not compatible etc.).
>
> WDYT? Are there any DaskExecutor users around or someone you know of who
> would rely on it heavily? Are there any reasons why we should keep it as is
> ? Which of the above scenarios would you like to see?
>
> I would love to hear your opinions on that. If we see that we would like to
> follow 1) or 2) I am happy to pass the message to the Dask community and
> let them know that they will need to be involved in case they will want to
> bring back Dask from suspension or removal - I did that before for some of
> the past issues and I am happy to be the messenger (hopefull no-one shoots
> me).
>
> J.
>

Re: [DISCUSS] Suspend (Remove?) Daskexecutor provider

[Hussein Awala <hu...@awala.fr>]

Since this executor was part of the Airflow core a few months ago, should
we maintain security patches for a reasonable period after officially
announcing its suspension?

On Wed, Nov 15, 2023 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hello Everyone,
>
> I would like to start discussion about potential suspension and eventually
> removal of Daskexecutor provider.
>
> After some recent changes and moving DaskExecutor from core to provider we
> have now an open option to suspend and eventually remove Dask Executor from
> our codebase.
>
> After we have the process tested and tried with Qubole, for me this is the
> next candidate to remove and get rid of some burden it involves.
>
> I believe this executor is hardly used by anyone - IMHO it does not really
> add a value to our Executor's offering and it causes a number of CI/
> maintenance issues for us, while we have little to no experience and
> incentive and knowledge to manage it. It slows us down, makes our CI flaky,
> sometimes delays our release process etc etc.
>
> Some of the past discussions about it (this is the third time I am (re)
> starting this discussion - the first time it was in 2020 when we discussed
> removing it for Airflow 2.0.
>
> * https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m
> * https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh
>
> Some of the recent problems with Dask executor impacting stability of our
> CI / tests/ impacting release process are manifested in those issues and
> PRs over the last few years (and this is just a subset of issues and PRs
> dealing with the stability of it.
>
> * https://github.com/apache/airflow/pull/35659
> * https://github.com/apache/airflow/issues/32778
> * https://github.com/apache/airflow/pull/35473
> * https://github.com/apache/airflow/pull/32991
> * https://github.com/apache/airflow/pull/30258
> * https://github.com/apache/airflow/pull/22046
> * https://github.com/apache/airflow/pull/22017
>
>
> Why now? What changed?
>
> One of the fantastic properties of the "providers" approach and moving the
> executor from core to provider - we can now ACTUALLY remove DaskExecutor
> without breaking any of our SemVer promises. This is something I was
> working on for quite a while and with some of the recent changes - making
> our Executor API stable and part of our public interface, introducing full
> life-cycle process for providers this is now entirely possible to remove
> all the maintenance burden while keeping our SemVer promises - so that we
> can do it without bumping Airflow to version 3.
>
> How does it work ? Simply - provider will remain released, and it will
> remain working in the version it has been released for the last time with
> all the future versions of Airflow, But at the same time we can remove the
> code from main of Airflow, stop running tests, stop caring about the
> provider impacting the constraints and generally speaking.
>
> If we decide to do somethin about, we have three options:
>
> 1) Suspend the daskexecutor provider
> 2) Remove it
> 3) We can also follow Suspend -> wait few months -> remove) pattern
>
> All of them are easy, all of them bring us much welcome relief in our
> release and CI and maintenance burden. All of them are well documented and
> tried and all that we need is just to agree which scenario we want to
> follow.
>
> In case 1) we will stop releasing new versions, but in case any new
> "development" is done on the dask side that will break currently released
> providers, it will be on someone who will take on the task to bring it back
> from suspension - fix all dependency issues, make sure tests are green and
> then it's just a matter of PR to bring it back. This is really a way to say
> "Hey Dask Executor community, if you want to keep any future development,
> it's on you to bring it back from suspension and take the task to solve all
> the difficulties we currently have to leave with in Airflow community".
>
> In case 2) we deliberately decide we do not want to maintain it any more
> and if someone would like to bring it back, they will have to convince the
> community and get the VOTE or LAZY CONSENSUS to bring it back (and then it
> could be even different provider package, not compatible etc.).
>
> WDYT? Are there any DaskExecutor users around or someone you know of who
> would rely on it heavily? Are there any reasons why we should keep it as is
> ? Which of the above scenarios would you like to see?
>
> I would love to hear your opinions on that. If we see that we would like to
> follow 1) or 2) I am happy to pass the message to the Dask community and
> let them know that they will need to be involved in case they will want to
> bring back Dask from suspension or removal - I did that before for some of
> the past issues and I am happy to be the messenger (hopefull no-one shoots
> me).
>
> J.
>