You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "eladkal (via GitHub)" <gi...@apache.org> on 2023/08/30 12:02:23 UTC

[GitHub] [airflow] eladkal opened a new issue, #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers

eladkal opened a new issue, #33916:
URL: https://github.com/apache/airflow/issues/33916

   ### Body
   
   Now that we have Executors in providers we should migrate
   https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery.html
   https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery_kubernetes.html
   https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/dask.html
   https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/kubernetes.html
   https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local_kubernetes.html
   
   To providers docs
   
   and update references
   https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#executor-types to the providers docs.
   
   We need also to update redirect.txt to reflect these changes
   
   
   ### Committer
   
   - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1699598249

   I don't see how this issue contradict that.
   
   The goal here is to make Executor be the same as Notifiers:
   https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html
   
   We have information about it in core and for the detailed info we refer to providers.
   
   The concept where code sits in one package yet documentation in another one is wrong to my taste.
   1. It creates confusion.
   2. When code is updated documentation will not follow.
   
   
   I don't think that things like how to configure Celery queue or info about git-sync of dag files to K8s workers should stay in airflow core docs. What is the motivation for this? Airflow core should have high level details - possibly a compare of capabilities between executors - anything that is higher level than a specific executor.
   
   If you still believe that this contradict the accepted vote I'd be happy to start a new discussion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1711928183

   Mailing list thread: [[DISCUSS] Executors docs should be published in Airflow core or providers?
   ](https://lists.apache.org/thread/vlhjgj3m7qzzcfdqwfrjkod8235sjzvp)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Migrate Celery/Kubernetes Executor docs to providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33916: Migrate Celery/Kubernetes Executor docs to providers
URL: https://github.com/apache/airflow/issues/33916


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Migrate Celery/Kubernetes Executor docs to providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1859262961

   Removed Dask from the list.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1731934897

   Vote is completed - [[LAZY CONSENSUS] Executors docs should be published in providers
   ](https://lists.apache.org/thread/xmqzqzk90v0wyj5crm191wsjh8y1q2r2)
   
   This task is now actionable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1699746052

   no argue from me :)
   I will start a mailing list thread to discuss clarification for that


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Migrate Celery/Dask/Kubernetes Executor docs to providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers
URL: https://github.com/apache/airflow/issues/33916


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Migrate Celery/Dask/Kubernetes Executor docs to providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1812508976

   This issue has been completed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Migrate Celery/Dask/Kubernetes Executor docs to providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1812509557

   Ah no :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1699729336

   It does contradict it. See the discussion and the comment that was specifically made in the comment:
   
   > Also, resulting from the discussion we will keep documentation for
   available executors in Airflow (so they will still be considered as THE
   executors available and will be discoverable in the same way as today).
   
   There were voices in the discussion that it would make it "harder" discoverable about the executors that we already had. So this statement was part of the "we are voting on it". I am not arguing whether it is good or bad. Maybe we should change the decision made then. But the "Vote" clearly included "Leaving documentation about former core execution in the core airflow documentation" and in order to move the docuementation to the providers we need to change that decision. That's all I am saying.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #33916: Migrate Celery/Dask/Kubernetes Executor docs to providers

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33916:
URL: https://github.com/apache/airflow/issues/33916#issuecomment-1699535538

   This is not what we agreed on during the vote.
   
   If we want to change it - this needs to be discussed at the devlist. 
   
   In the discussion we agreed (and we deliberately left)  the "core-concepts" executor documentation  as part of airflow. While I think the code should be separated. the description of possible executors (the ones that used to be "core" ) should stay in Airflow Documentation for quite a long time (maybe forever).
   
   
   This is what has been voted on: 
   
   > Also, resulting from the discussion we will keep documentation for
   available executors in Airflow (so they will still be considered as THE
   executors available and will be discoverable in the same way as today).
   
   Vote thread here: https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org