You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "eladkal (via GitHub)" <gi...@apache.org> on 2023/08/30 12:02:23 UTC
[GitHub] [airflow] eladkal opened a new issue, #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers
eladkal opened a new issue, #33916:
URL: https://github.com/apache/airflow/issues/33916
### Body
Now that we have Executors in providers we should migrate
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery.html
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery_kubernetes.html
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/dask.html
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/kubernetes.html
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local_kubernetes.html
To providers docs
and update references
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#executor-types to the providers docs.
We need also to update redirect.txt to reflect these changes
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1699598249
I don't see how this issue contradict that.
The goal here is to make Executor be the same as Notifiers:
https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html
We have information about it in core and for the detailed info we refer to providers.
The concept where code sits in one package yet documentation in another one is wrong to my taste.
1. It creates confusion.
2. When code is updated documentation will not follow.
I don't think that things like how to configure Celery queue or info about git-sync of dag files to K8s workers should stay in airflow core docs. What is the motivation for this? Airflow core should have high level details - possibly a compare of capabilities between executors - anything that is higher level than a specific executor.
If you still believe that this contradict the accepted vote I'd be happy to start a new discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1711928183
Mailing list thread: [[DISCUSS] Executors docs should be published in Airflow core or providers?
](https://lists.apache.org/thread/vlhjgj3m7qzzcfdqwfrjkod8235sjzvp)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Migrate Celery/Kubernetes Executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33916: Migrate Celery/Kubernetes Executor docs to providers
URL: https://github.com/apache/airflow/issues/33916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Migrate Celery/Kubernetes Executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1859262961
Removed Dask from the list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1731934897
Vote is completed - [[LAZY CONSENSUS] Executors docs should be published in providers
](https://lists.apache.org/thread/xmqzqzk90v0wyj5crm191wsjh8y1q2r2)
This task is now actionable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1699746052
no argue from me :)
I will start a mailing list thread to discuss clarification for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Migrate Celery/Dask/Kubernetes Executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers
URL: https://github.com/apache/airflow/issues/33916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Migrate Celery/Dask/Kubernetes Executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1812508976
This issue has been completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Migrate Celery/Dask/Kubernetes Executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1812509557
Ah no :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1699729336
It does contradict it. See the discussion and the comment that was specifically made in the comment:
> Also, resulting from the discussion we will keep documentation for
available executors in Airflow (so they will still be considered as THE
executors available and will be discoverable in the same way as today).
There were voices in the discussion that it would make it "harder" discoverable about the executors that we already had. So this statement was part of the "we are voting on it". I am not arguing whether it is good or bad. Maybe we should change the decision made then. But the "Vote" clearly included "Leaving documentation about former core execution in the core airflow documentation" and in order to move the docuementation to the providers we need to change that decision. That's all I am saying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1699535538
This is not what we agreed on during the vote.
If we want to change it - this needs to be discussed at the devlist.
In the discussion we agreed (and we deliberately left) the "core-concepts" executor documentation as part of airflow. While I think the code should be separated. the description of possible executors (the ones that used to be "core" ) should stay in Airflow Documentation for quite a long time (maybe forever).
This is what has been voted on:
> Also, resulting from the discussion we will keep documentation for
available executors in Airflow (so they will still be considered as THE
executors available and will be discoverable in the same way as today).
Vote thread here: https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org