You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Elad Kalif <el...@apache.org> on 2023/09/08 16:14:44 UTC
[DISCUSS] Executors docs should be published in Airflow core or providers?
Hello everyone,
This thread is opened due to open issue Migrate Celery/Dask/Kubernetes
Executor docs to providers <https://github.com/apache/airflow/issues/33916>
*Background:*
We had a discussion about extracting Celery, Kubernetes, Dask executors
from core to providers (discussion thread
<https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do>, vote
thread <https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8>)
One of the things we voted on was:
Also, resulting from the discussion we will keep documentation for
> available executors in Airflow (so they will still be considered as THE
> executors available and will be discoverable in the same way as today).
*The problem:*
Airflow Core and Providers do not share the same release cycle nor cadance.
This means that if we add new capabilities to executors or fix an issue
which requires both code and doc update - the code will be delivered
before/ahead of documentation. Both cases are not good.
*My proposal:*Now that Celery, Kubernetes, Dask executors are in providers.
The section of core-concept
<https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html>
should
contain only HIGH level information about the executors. It should not
contain information about executors internals or how to address common
problems. This information should be in the provider docs. The high level
info should be short and on a level that is relevant for all executors
which means it's not likely to have many changes over time. The
core-concept should have links/refers for deep dive read to the provider
docs. This is very similar to what we do with Notifiers
<https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html>
core
contains high level information and a list of notifers that are linked to
provider docs.
WDYT?
Also, resulting from the discussion we will keep documentation for
available executors in Airflow (so they will still be considered as THE
executors available and will be discoverable in the same way as today).Moe
K8S and Celery Executors (and related) to respective providers?
Re: [DISCUSS] Executors docs should be published in Airflow core or providers?
Posted by Elad Kalif <el...@apache.org>.
as there were no concerns raised I started a lazy consensus
https://lists.apache.org/thread/xmqzqzk90v0wyj5crm191wsjh8y1q2r2
On Mon, Sep 11, 2023 at 4:15 PM Oliveira, Niko <on...@amazon.com.invalid>
wrote:
> +1 to this!
>
> I also have a docs section half written on the executor interface and how
> to extend it. But I've been very busy with a few other items that are
> completing soon.
>
> Cheers,
> Niko
>
> ________________________________
> From: Pankaj Koti <pa...@astronomer.io.INVALID>
> Sent: Friday, September 8, 2023 12:03:35 PM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Executors docs should
> be published in Airflow core or providers?
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> +1 to the proposal.
>
> I think core Airflow docs can contain details about the default executor
> that
> gets shipped with standalone Airflow installation and a short note about
> possibilities of using other (providers) executors in production and saying
> to look
> for detailed docs in the corresponding provider.
>
> Regards,
>
>
>
> Pankaj Koti
>
> *Senior Software Engineer, *OSS Engineering Team.
> Location: Pune, India
>
> Timezone: Indian Standard Time (IST)
>
> Email: pankaj.koti@astronomer.io
>
> Mobile: +91 9730079985
>
>
> On Fri, Sep 8, 2023 at 11:28 PM Hussein Awala <hu...@awala.fr> wrote:
>
> > Since we moved the executors to the providers packages and made the
> > executor interface pluggable and extensible, we should move the docs to
> > their corresponding providers. However, we need to keep a doc in Airflow
> > core that explains how to use/configure a provider executor (as we have
> for
> > the secret managers and the task log handlers) and maybe how to create a
> > new custom one.
> >
> > On Fri, Sep 8, 2023 at 6:15 PM Elad Kalif <el...@apache.org> wrote:
> >
> > > Hello everyone,
> > >
> > > This thread is opened due to open issue Migrate Celery/Dask/Kubernetes
> > > Executor docs to providers <
> > https://github.com/apache/airflow/issues/33916
> > > >
> > >
> > > *Background:*
> > > We had a discussion about extracting Celery, Kubernetes, Dask executors
> > > from core to providers (discussion thread
> > > <https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do>,
> vote
> > > thread <
> https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
> > >)
> > >
> > > One of the things we voted on was:
> > >
> > > Also, resulting from the discussion we will keep documentation for
> > > > available executors in Airflow (so they will still be considered as
> THE
> > > > executors available and will be discoverable in the same way as
> today).
> > >
> > >
> > > *The problem:*
> > > Airflow Core and Providers do not share the same release cycle nor
> > cadance.
> > > This means that if we add new capabilities to executors or fix an issue
> > > which requires both code and doc update - the code will be delivered
> > > before/ahead of documentation. Both cases are not good.
> > >
> > >
> > > *My proposal:*Now that Celery, Kubernetes, Dask executors are in
> > providers.
> > > The section of core-concept
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html
> > > >
> > > should
> > > contain only HIGH level information about the executors. It should not
> > > contain information about executors internals or how to address common
> > > problems. This information should be in the provider docs. The high
> level
> > > info should be short and on a level that is relevant for all executors
> > > which means it's not likely to have many changes over time. The
> > > core-concept should have links/refers for deep dive read to the
> provider
> > > docs. This is very similar to what we do with Notifiers
> > > <
> > >
> >
> https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html
> > > >
> > > core
> > > contains high level information and a list of notifers that are linked
> to
> > > provider docs.
> > >
> > > WDYT?
> > >
> > > Also, resulting from the discussion we will keep documentation for
> > > available executors in Airflow (so they will still be considered as THE
> > > executors available and will be discoverable in the same way as
> > today).Moe
> > > K8S and Celery Executors (and related) to respective providers?
> > >
> >
>
Re: [DISCUSS] Executors docs should be published in Airflow core or providers?
Posted by "Oliveira, Niko" <on...@amazon.com.INVALID>.
+1 to this!
I also have a docs section half written on the executor interface and how to extend it. But I've been very busy with a few other items that are completing soon.
Cheers,
Niko
________________________________
From: Pankaj Koti <pa...@astronomer.io.INVALID>
Sent: Friday, September 8, 2023 12:03:35 PM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Executors docs should be published in Airflow core or providers?
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
+1 to the proposal.
I think core Airflow docs can contain details about the default executor
that
gets shipped with standalone Airflow installation and a short note about
possibilities of using other (providers) executors in production and saying
to look
for detailed docs in the corresponding provider.
Regards,
Pankaj Koti
*Senior Software Engineer, *OSS Engineering Team.
Location: Pune, India
Timezone: Indian Standard Time (IST)
Email: pankaj.koti@astronomer.io
Mobile: +91 9730079985
On Fri, Sep 8, 2023 at 11:28 PM Hussein Awala <hu...@awala.fr> wrote:
> Since we moved the executors to the providers packages and made the
> executor interface pluggable and extensible, we should move the docs to
> their corresponding providers. However, we need to keep a doc in Airflow
> core that explains how to use/configure a provider executor (as we have for
> the secret managers and the task log handlers) and maybe how to create a
> new custom one.
>
> On Fri, Sep 8, 2023 at 6:15 PM Elad Kalif <el...@apache.org> wrote:
>
> > Hello everyone,
> >
> > This thread is opened due to open issue Migrate Celery/Dask/Kubernetes
> > Executor docs to providers <
> https://github.com/apache/airflow/issues/33916
> > >
> >
> > *Background:*
> > We had a discussion about extracting Celery, Kubernetes, Dask executors
> > from core to providers (discussion thread
> > <https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do>, vote
> > thread <https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
> >)
> >
> > One of the things we voted on was:
> >
> > Also, resulting from the discussion we will keep documentation for
> > > available executors in Airflow (so they will still be considered as THE
> > > executors available and will be discoverable in the same way as today).
> >
> >
> > *The problem:*
> > Airflow Core and Providers do not share the same release cycle nor
> cadance.
> > This means that if we add new capabilities to executors or fix an issue
> > which requires both code and doc update - the code will be delivered
> > before/ahead of documentation. Both cases are not good.
> >
> >
> > *My proposal:*Now that Celery, Kubernetes, Dask executors are in
> providers.
> > The section of core-concept
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html
> > >
> > should
> > contain only HIGH level information about the executors. It should not
> > contain information about executors internals or how to address common
> > problems. This information should be in the provider docs. The high level
> > info should be short and on a level that is relevant for all executors
> > which means it's not likely to have many changes over time. The
> > core-concept should have links/refers for deep dive read to the provider
> > docs. This is very similar to what we do with Notifiers
> > <
> >
> https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html
> > >
> > core
> > contains high level information and a list of notifers that are linked to
> > provider docs.
> >
> > WDYT?
> >
> > Also, resulting from the discussion we will keep documentation for
> > available executors in Airflow (so they will still be considered as THE
> > executors available and will be discoverable in the same way as
> today).Moe
> > K8S and Celery Executors (and related) to respective providers?
> >
>
Re: [DISCUSS] Executors docs should be published in Airflow core or providers?
Posted by Pankaj Koti <pa...@astronomer.io.INVALID>.
+1 to the proposal.
I think core Airflow docs can contain details about the default executor
that
gets shipped with standalone Airflow installation and a short note about
possibilities of using other (providers) executors in production and saying
to look
for detailed docs in the corresponding provider.
Regards,
Pankaj Koti
*Senior Software Engineer, *OSS Engineering Team.
Location: Pune, India
Timezone: Indian Standard Time (IST)
Email: pankaj.koti@astronomer.io
Mobile: +91 9730079985
On Fri, Sep 8, 2023 at 11:28 PM Hussein Awala <hu...@awala.fr> wrote:
> Since we moved the executors to the providers packages and made the
> executor interface pluggable and extensible, we should move the docs to
> their corresponding providers. However, we need to keep a doc in Airflow
> core that explains how to use/configure a provider executor (as we have for
> the secret managers and the task log handlers) and maybe how to create a
> new custom one.
>
> On Fri, Sep 8, 2023 at 6:15 PM Elad Kalif <el...@apache.org> wrote:
>
> > Hello everyone,
> >
> > This thread is opened due to open issue Migrate Celery/Dask/Kubernetes
> > Executor docs to providers <
> https://github.com/apache/airflow/issues/33916
> > >
> >
> > *Background:*
> > We had a discussion about extracting Celery, Kubernetes, Dask executors
> > from core to providers (discussion thread
> > <https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do>, vote
> > thread <https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
> >)
> >
> > One of the things we voted on was:
> >
> > Also, resulting from the discussion we will keep documentation for
> > > available executors in Airflow (so they will still be considered as THE
> > > executors available and will be discoverable in the same way as today).
> >
> >
> > *The problem:*
> > Airflow Core and Providers do not share the same release cycle nor
> cadance.
> > This means that if we add new capabilities to executors or fix an issue
> > which requires both code and doc update - the code will be delivered
> > before/ahead of documentation. Both cases are not good.
> >
> >
> > *My proposal:*Now that Celery, Kubernetes, Dask executors are in
> providers.
> > The section of core-concept
> > <
> >
> https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html
> > >
> > should
> > contain only HIGH level information about the executors. It should not
> > contain information about executors internals or how to address common
> > problems. This information should be in the provider docs. The high level
> > info should be short and on a level that is relevant for all executors
> > which means it's not likely to have many changes over time. The
> > core-concept should have links/refers for deep dive read to the provider
> > docs. This is very similar to what we do with Notifiers
> > <
> >
> https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html
> > >
> > core
> > contains high level information and a list of notifers that are linked to
> > provider docs.
> >
> > WDYT?
> >
> > Also, resulting from the discussion we will keep documentation for
> > available executors in Airflow (so they will still be considered as THE
> > executors available and will be discoverable in the same way as
> today).Moe
> > K8S and Celery Executors (and related) to respective providers?
> >
>
Re: [DISCUSS] Executors docs should be published in Airflow core or providers?
Posted by Hussein Awala <hu...@awala.fr>.
Since we moved the executors to the providers packages and made the
executor interface pluggable and extensible, we should move the docs to
their corresponding providers. However, we need to keep a doc in Airflow
core that explains how to use/configure a provider executor (as we have for
the secret managers and the task log handlers) and maybe how to create a
new custom one.
On Fri, Sep 8, 2023 at 6:15 PM Elad Kalif <el...@apache.org> wrote:
> Hello everyone,
>
> This thread is opened due to open issue Migrate Celery/Dask/Kubernetes
> Executor docs to providers <https://github.com/apache/airflow/issues/33916
> >
>
> *Background:*
> We had a discussion about extracting Celery, Kubernetes, Dask executors
> from core to providers (discussion thread
> <https://lists.apache.org/thread/kwwhz62lddygodpgr3fk4b9tthtld9do>, vote
> thread <https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8>)
>
> One of the things we voted on was:
>
> Also, resulting from the discussion we will keep documentation for
> > available executors in Airflow (so they will still be considered as THE
> > executors available and will be discoverable in the same way as today).
>
>
> *The problem:*
> Airflow Core and Providers do not share the same release cycle nor cadance.
> This means that if we add new capabilities to executors or fix an issue
> which requires both code and doc update - the code will be delivered
> before/ahead of documentation. Both cases are not good.
>
>
> *My proposal:*Now that Celery, Kubernetes, Dask executors are in providers.
> The section of core-concept
> <
> https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/index.html
> >
> should
> contain only HIGH level information about the executors. It should not
> contain information about executors internals or how to address common
> problems. This information should be in the provider docs. The high level
> info should be short and on a level that is relevant for all executors
> which means it's not likely to have many changes over time. The
> core-concept should have links/refers for deep dive read to the provider
> docs. This is very similar to what we do with Notifiers
> <
> https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html
> >
> core
> contains high level information and a list of notifers that are linked to
> provider docs.
>
> WDYT?
>
> Also, resulting from the discussion we will keep documentation for
> available executors in Airflow (so they will still be considered as THE
> executors available and will be discoverable in the same way as today).Moe
> K8S and Celery Executors (and related) to respective providers?
>
Re: [DISCUSS] Executors docs should be published in Airflow core or providers?
Posted by Aritra Basu <ar...@gmail.com>.
Sounds reasonable to me
--
Regards,
Aritra Basu
On Fri, Sep 8, 2023, 10:03 PM Vincent Beck <vi...@apache.org> wrote:
> +1
>
> On 2023/09/08 16:24:10 "Ferruzzi, Dennis" wrote:
> > I like it.
> >
> >
> > - ferruzzi
> >
> >
> > ________________________________
> > From: Jarek Potiuk <ja...@potiuk.com>
> > Sent: Friday, September 8, 2023 9:19 AM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Executors docs
> should be published in Airflow core or providers?
> >
> > CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> certain que le contenu ne présente aucun risque.
> >
> >
> >
> > +1
> >
> > On Fri, Sep 8, 2023 at 6:18 PM Daniel Standish
> > <da...@astronomer.io.invalid> wrote:
> >
> > > Sounds reasonable.
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Executors docs should be published in Airflow core or providers?
Posted by Vincent Beck <vi...@apache.org>.
+1
On 2023/09/08 16:24:10 "Ferruzzi, Dennis" wrote:
> I like it.
>
>
> - ferruzzi
>
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Friday, September 8, 2023 9:19 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Executors docs should be published in Airflow core or providers?
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
>
>
>
> +1
>
> On Fri, Sep 8, 2023 at 6:18 PM Daniel Standish
> <da...@astronomer.io.invalid> wrote:
>
> > Sounds reasonable.
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Executors docs should be published in Airflow core or providers?
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
I like it.
- ferruzzi
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Friday, September 8, 2023 9:19 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL] [COURRIEL EXTERNE] [DISCUSS] Executors docs should be published in Airflow core or providers?
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
+1
On Fri, Sep 8, 2023 at 6:18 PM Daniel Standish
<da...@astronomer.io.invalid> wrote:
> Sounds reasonable.
>
Re: [DISCUSS] Executors docs should be published in Airflow core or providers?
Posted by Jarek Potiuk <ja...@potiuk.com>.
+1
On Fri, Sep 8, 2023 at 6:18 PM Daniel Standish
<da...@astronomer.io.invalid> wrote:
> Sounds reasonable.
>
Re: [DISCUSS] Executors docs should be published in Airflow core or providers?
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
Sounds reasonable.