You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "RNHTTR (via GitHub)" <gi...@apache.org> on 2023/10/06 21:51:15 UTC
[PR] Migrate provider-specific executor docs to providers [airflow]
RNHTTR opened a new pull request, #34809:
URL: https://github.com/apache/airflow/pull/34809
Closes: #33916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #34809:
URL: https://github.com/apache/airflow/pull/34809#discussion_r1351443468
##########
docs/apache-airflow/administration-and-deployment/production-deployment.rst:
##########
@@ -60,8 +60,8 @@ Airflow uses :class:`~airflow.executors.sequential_executor.SequentialExecutor`
nature, the user is limited to executing at most one task at a time. ``Sequential Executor`` also pauses
the scheduler when it runs a task, hence it is not recommended in a production setup. You should use the
:class:`~airflow.executors.local_executor.LocalExecutor` for a single machine.
-For a multi-node setup, you should use the :doc:`Kubernetes executor <../core-concepts/executor/kubernetes>` or
-the :doc:`Celery executor <../core-concepts/executor/celery>`.
+For a multi-node setup, you should use the :doc:`Kubernetes executor <.apache-airflow-providers-cncf-kubernetes:kubernetes_executor>` or
Review Comment:
```suggestion
For a multi-node setup, you should use the :doc:`Kubernetes executor <apache-airflow-providers-cncf-kubernetes:kubernetes_executor>` or
```
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -51,28 +51,27 @@ If you want to check which executor is currently set, you can use the ``airflow
Executor Types
--------------
-There are two types of executor - those that run tasks *locally* (inside the ``scheduler`` process), and those that run their tasks *remotely* (usually via a pool of *workers*). Airflow comes configured with the ``SequentialExecutor`` by default, which is a local executor, and the safest option for execution, but we *strongly recommend* you change this to ``LocalExecutor`` for small, single-machine installations, or one of the remote executors for a multi-machine/cloud installation.
+Executors are primarily responsible for determining where :doc:`task instances </core-concepts/tasks>` are executed, as well as managing the state of task instances. Executors have a common API and are "pluggable", meaning you can swap executors based on your installation needs.
**Local Executors**
.. toctree::
:maxdepth: 1
- debug
local
sequential
-**Remote Executors**
+The DebugExecutor also exists but has been deprecated in favor of using the ``dag.test()`` command for :doc:`debugging <../debug>`.
-.. toctree::
- :maxdepth: 1
- celery
- celery_kubernetes
- dask
- kubernetes
- local_kubernetes
+**Remote Executors**
+
+* :doc:`CeleryExecutor apache-airflow-providers-celery:celery_executor`
+* :doc:`CeleryKubernetesExecutor apache-airflow-providers-celery:celery_kubernetes_executor`
+* :doc:`DaskExecutor apache-airflow-providers-daskexecutor:dask_executor`
Review Comment:
```suggestion
* :doc:`CeleryExecutor <apache-airflow-providers-celery:celery_executor>`
* :doc:`CeleryKubernetesExecutor <apache-airflow-providers-celery:celery_kubernetes_executor>`
* :doc:`DaskExecutor <apache-airflow-providers-daskexecutor:dask_executor>`
```
[Maybe like this example](https://github.com/apache/airflow/blob/9a297380558420cc2f73a409e350f333530e279e/docs/apache-airflow/configurations-ref.rst?plain=1#L38)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1774149040
> Strange...
>
> ```
> breeze build-docs --docs-only
> ```
>
> This builds successfully, but if I remove `--docs-only`, the build fails...
You can use `--clean-build` flag - likely this one will reveal the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1775614880
Yep. Looks better. I think those are not spellcheck errors but some regular sphinx riddiles. Sphinx does not seem to like SOMETHING in the docs and not tell exactly where. The usual thing which hopefully when we move to #33156 might get better. For now it requires a bit guessing what's wrong, unfortunately.
```
############################## Start docs build errors summary ##############################
============================== apache-airflow-providers-celery ==============================
------------------------------ Error 1 --------------------
WARNING: undefined label: 'executor:celeryexecutor'
File path: apache-airflow-providers-celery/_api/airflow/providers/celery/executors/celery_executor/index.rst (4)
1 | :py:mod:`airflow.providers.celery.executors.celery_executor`
2 | ============================================================
3 |
> 4 | .. py:module:: airflow.providers.celery.executors.celery_executor
5 |
6 | .. autoapi-nested-parse::
7 |
8 | CeleryExecutor.
9 |
============================== apache-airflow-providers-cncf-kubernetes ==============================
------------------------------ Error 1 --------------------
WARNING: undefined label: 'executor:kubernetesexecutor'
File path: apache-airflow-providers-cncf-kubernetes/_api/airflow/providers/cncf/kubernetes/executors/kubernetes_executor/index.rst (4)
1 | :py:mod:`airflow.providers.cncf.kubernetes.executors.kubernetes_executor`
2 | =========================================================================
3 |
> 4 | .. py:module:: airflow.providers.cncf.kubernetes.executors.kubernetes_executor
5 |
6 | .. autoapi-nested-parse::
7 |
8 | KubernetesExecutor.
9 |
============================== apache-airflow-providers-daskexecutor ==============================
------------------------------ Error 1 --------------------
WARNING: undefined label: 'executor:daskexecutor'
File path: apache-airflow-providers-daskexecutor/_api/airflow/providers/daskexecutor/executors/dask_executor/index.rst (4)
1 | :py:mod:`airflow.providers.daskexecutor.executors.dask_executor`
2 | ================================================================
3 |
> 4 | .. py:module:: airflow.providers.daskexecutor.executors.dask_executor
5 |
6 | .. autoapi-nested-parse::
7 |
8 | DaskExecutor.
9 |
############################## End docs build errors summary ##############################
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1773817679
> Is this just a flaky test? It builds fine locally
Likely one more fix to recently completed `docs-build` command for selective docs building - let me see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1848964051
> @lzdanski This is the spell check sphinx issue I was referring to.
@RNHTTR - I think first thing to do is to rebase this one. It's quite far behind, we introduced a number of changes (for example we split docs building from spellcheck) - maybe after rebase things will get clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1774156417
Guessing what could happen here (and those are just guesses looking at the output)
I think the problem is that you have changes in both "airflow" and "celery" and other packages that refer to each other in both directions.
The way how docs building is done is that it builds each package separately and uses the "inventory" from other packages (downloaded from s3) it refers in order to verify if the document exist. Once the package successfully builds the docs, the downloaded inventory gets updated locally.
In most cases where there are two packages and only one of them refer to the other document, this document building has multi-pass implemented. If the package fails to be built it is added to the queue and retried again (up to three times in case the dependencies are A -> B -> C (so if A links to B and B links to C in the first pass C succeeds, in the second B will succeed as it will use locally built inventory from C and in the third A will finaly succeed using the locally built inventory from B.
The problem is that if those packages are build together and they are referring to each other's new documents - none of them can succed - because they are referring to each other's documents, and those documents do not exist in the remote inventory.
Looking at the output - it almost worked:
You can see that at first pass airflow + cncf.kubernetes + celery + dask failed docs building
At the second pass the three others succeeded, only airflow was left.
Unfortunately the third pass on Airflow failed.
The error you see:
Is that API documents could not be found in Airflow - most likely because the "clean" step deleted it and for some reason they were not recreated by the API plugin. So maybe we can attempt to remove the clean step between the retries.
Suggestion how to fix it (if my guess is right):
Only clean the docs when you start and not when you retry. Might be a good exercise to learn how the build process works.
As i understand it (again by looking at the code - I modified it quite a few times but mostly when I saw similar issues):
In build_docs.py :
The `build_docs_for_packages` is executed several times to make the multi-pass works
* The first time it should be called with parameter to run `cleanup`
* all the other times `retry_building_docs_if_needed` it should be called without running cleanup
That could probably solve the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1764481201
> Yeah, I see lots of references from `apache-airflow` to `apache-airflow-providers` (for example, [this one](https://github.com/apache/airflow/blob/main/docs/apache-airflow/operators-and-hooks-ref.rst?plain=1#L26)), but when I try to do `` :doc:`apache-airflow-providers-celery:celery_executor` ``, I get the following error during the build:
>
>
> `celery_executor.rst` now lives at `docs/apache-airflow-providers-celery/celery_executor.rst`.
yes. Sphinx is picky and requires sometimes a lot of persistence and trying out things looking at other examples.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1965353334
Closing this in favor of #37728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1751443900
It's currently unclear why I'm able to reference ``:doc:`apache-airflow:<some document in core docs>` ``, but I can't seem to link, for example, ``:doc:`apache-airflow-providers-celery:celery_executor` ``
I'd appreciate if any reStructuredTexperts™️ / Sphinx experts know how to link from the `apache-airflow` docs to a provider doc :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1774310032
That explains why a build would succeed after a handful of failure messages :) Thanks, @potiuk .
I added a param to not perform the clean step if it's on a retried build. Still getting empty spell check errors that I need to investigate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1751738799
Yeah, I see lots of references from `apache-airflow` to `apache-airflow-providers`, but when I try to do ``:doc:`apache-airflow-providers-celery:celery_executor` ``, I get the following error during the build:
```
------------------------------ Error 5 --------------------
WARNING: unknown document: 'CeleryExecutor apache-airflow-providers-celery:celery_executor'
File path: apache-airflow/core-concepts/executor/index.rst (70)
65 | The DebugExecutor also exists but has been deprecated in favor of using the ``dag.test()`` command for :doc:`debugging <../debug>`.
66 |
67 |
68 | **Remote Executors**
69 |
> 70 | * :doc:`CeleryExecutor apache-airflow-providers-celery:celery_executor`
71 | * :doc:`CeleryKubernetesExecutor apache-airflow-providers-celery:celery_kubernetes_executor`
72 | * :doc:`DaskExecutor apache-airflow-providers-daskexecutor:dask_executor`
73 | * :doc:`KubernetesExecutor <apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
74 | * :doc:`KubernetesLocalExecutor <apache-airflow-providers-cncf-kubernetes:local_kubernetes_executor>`
75 |
```
`celery_executor.rst` now lives at `docs/apache-airflow-providers-celery/celery_executor.rst`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1773892784
> Thanks. What's the best way for me to retry the failed test?
Just did. Lets see
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1774094261
Strange...
```
breeze build-docs --docs-only
```
This builds successfully, but if I remove `--docs-only`, the build fails...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1963202043
Can focus on this again -- will mark ready for review when tests are passing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1891503309
@RNHTTR can you rebase and resolve conflicts? I see the current failure is on openai provider which is odd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1908962757
Just to comment on the errors @RNHTTR and guide you a bit..
I think all the _api errors are coming from those first few errors here. Usually they appear when the main package failed to build even after multiple attempts. They are really side-effects of the **real**l problems:
Error 1) in openai -> seems that there is extra file added
Error 1) Error 2) in apache-airflow -> I thin the references are just pointing to the moved docs - they should point to the new location. I think those 2 are preventing the `apache-airflow` package dogs from being built (and eventually trigger the API errors)
You should track the others :)
```
============================== apache-airflow-providers-openai ==============================
------------------------------ Error 1 --------------------
/opt/airflow/docs/apache-airflow-providers-openai/dask_executor.rst: WARNING: document isn't included in any toctree
============================== apache-airflow ==============================
------------------------------ Error 1 --------------------
WARNING: unknown document: '../core-concepts/executor/celery'
File path: apache-airflow/administration-and-deployment/production-deployment.rst (173)
168 | scheduler and you cannot upgrade the Scheduler without killing the tasks run by it. You can either
169 | pause all your DAGs and wait for the running tasks to complete or just stop the scheduler and kill all
170 | the tasks it runs - then you will need to clear and restart those tasks manually after the upgrade
171 | is completed (or rely on ``retry`` being set for stopped tasks).
172 |
>173 | * For the :doc:`Celery executor <../core-concepts/executor/celery>`, you have to first put your workers in
174 | offline mode (usually by setting a single ``TERM`` signal to the workers), wait until the workers
175 | finish all the running tasks, and then upgrade the code (for example by replacing the image the workers run
176 | in and restart the workers). You can monitor your workers via ``flower`` monitoring tool and see the number
177 | of running tasks going down to zero. Once the workers are upgraded, they will be automatically put in online
178 | mode and start picking up new tasks. You can then upgrade the ``Scheduler`` in a rolling restart mode.
------------------------------ Error 2 --------------------
WARNING: unknown document: '../core-concepts/executor/kubernetes'
File path: apache-airflow/administration-and-deployment/production-deployment.rst (180)
175 | finish all the running tasks, and then upgrade the code (for example by replacing the image the workers run
176 | in and restart the workers). You can monitor your workers via ``flower`` monitoring tool and see the number
177 | of running tasks going down to zero. Once the workers are upgraded, they will be automatically put in online
178 | mode and start picking up new tasks. You can then upgrade the ``Scheduler`` in a rolling restart mode.
179 |
>180 | * For the :doc:`Kubernetes executor <../core-concepts/executor/kubernetes>`, you can upgrade the scheduler
181 | triggerer, webserver in a rolling restart mode, and generally you should not worry about the workers, as they
182 | are managed by the Kubernetes cluster and will be automatically adopted by ``Schedulers`` when they are
183 | upgraded and restarted.
184 |
185 | * For the :doc:``CeleryKubernetesExecutor <../core-concepts/executor/celery-kubernetes>``, you follow the
------------------------------ Error 3 --------------------
WARNING: unknown document: 'apache-airflow-providers-daskexecutor:dask_executor'
File path: apache-airflow/core-concepts/executor/index.rst (65)
60 |
61 | **Remote Executors**
62 |
63 | * :doc:`CeleryExecutor <apache-airflow-providers-celery:celery_executor>`
64 | * :doc:`CeleryKubernetesExecutor <apache-airflow-providers-celery:celery_kubernetes_executor>`
> 65 | * :doc:`DaskExecutor <apache-airflow-providers-daskexecutor:dask_executor>`
66 | * :doc:`KubernetesExecutor <apache-airflow-providers-cncf-kubernetes:kubernetes_executor>`
67 | * :doc:`KubernetesLocalExecutor <apache-airflow-providers-cncf-kubernetes:local_kubernetes_executor>`
68 |
69 |
70 | .. note::
------------------------------ Error 4 --------------------
WARNING: unknown document: 'executor/celery'
File path: apache-airflow/core-concepts/overview.rst (68)
63 | ...................
64 |
65 | Some Airflow components are optional and can enable better extensibility, scalability, and
66 | performance in your Airflow:
67 |
> 68 | * Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
69 | worker might be part of the scheduler not a separate component. It can be run as a long running process
70 | in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
71 | :doc:`KubernetesExecutor <executor/kubernetes>`.
72 |
73 | * Optional *triggerer*, which executes deferred tasks in an asyncio event loop. In basic installation
------------------------------ Error 5 --------------------
WARNING: unknown document: 'executor/kubernetes'
File path: apache-airflow/core-concepts/overview.rst (68)
63 | ...................
64 |
65 | Some Airflow components are optional and can enable better extensibility, scalability, and
66 | performance in your Airflow:
67 |
> 68 | * Optional *worker*, which executes the tasks given to it by the scheduler. In the basic installation
69 | worker might be part of the scheduler not a separate component. It can be run as a long running process
70 | in the :doc:`CeleryExecutor <executor/celery>`, or as a POD in the
71 | :doc:`KubernetesExecutor <executor/kubernetes>`.
72 |
73 | * Optional *triggerer*, which executes deferred tasks in an asyncio event loop.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1751656980
> It's currently unclear why I'm able to reference `` :doc:`apache-airflow:<some document in core docs>` ``, but I can't seem to link, for example, `` :doc:`apache-airflow-providers-celery:celery_executor` ``
>
> I'd appreciate if any reStructuredTexperts™️ / Sphinx experts know how to link from the `apache-airflow` docs to a provider doc :D
If not mistaken we have example in Notifer doc that links core to provider docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1815015960
> Ho about completing it :)?
Been traveling / busy / sick the last few weeks. Hoping to pick this back up next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1773929085
I've added `full tests needed label adn rebased it -> likely the issues are because only subset of docs was built due to selective checks. With `full tests needed` label, all of them will run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1775697475
Locally, I get the following ☹️
```
The documentation has errors.
#################### Packages errors summary ####################
Package name Count of doc build errors Count of spelling errors
------------------------------------------------- --------------------------- --------------------------
apache-airflow-providers-celery 0 1
apache-airflow-providers-cncf-kubernetes 0 1
apache-airflow-providers-daskexecutor 0 1
##################################################
```
I'll keep digging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on code in PR #34809:
URL: https://github.com/apache/airflow/pull/34809#discussion_r1361422911
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -51,28 +51,27 @@ If you want to check which executor is currently set, you can use the ``airflow
Executor Types
--------------
-There are two types of executor - those that run tasks *locally* (inside the ``scheduler`` process), and those that run their tasks *remotely* (usually via a pool of *workers*). Airflow comes configured with the ``SequentialExecutor`` by default, which is a local executor, and the safest option for execution, but we *strongly recommend* you change this to ``LocalExecutor`` for small, single-machine installations, or one of the remote executors for a multi-machine/cloud installation.
+Executors are primarily responsible for determining where :doc:`task instances </core-concepts/tasks>` are executed, as well as managing the state of task instances. Executors have a common API and are "pluggable", meaning you can swap executors based on your installation needs.
**Local Executors**
.. toctree::
:maxdepth: 1
- debug
local
sequential
-**Remote Executors**
+The DebugExecutor also exists but has been deprecated in favor of using the ``dag.test()`` command for :doc:`debugging <../debug>`.
-.. toctree::
- :maxdepth: 1
- celery
- celery_kubernetes
- dask
- kubernetes
- local_kubernetes
+**Remote Executors**
+
+* :doc:`CeleryExecutor apache-airflow-providers-celery:celery_executor`
+* :doc:`CeleryKubernetesExecutor apache-airflow-providers-celery:celery_kubernetes_executor`
+* :doc:`DaskExecutor apache-airflow-providers-daskexecutor:dask_executor`
Review Comment:
jeez, thanks. I guess my eyes glazed over after too much rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1766881236
We need also to edit [docs/apache-airflow/redirects.txt](https://github.com/apache/airflow/blob/faa8f543d5f7bf7113076dab5f579493ec2860ec/docs/apache-airflow/redirects.txt)
This will allow referencing from old paths to the new paths without getting 404 page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1773811868
Is this just a flaky test? It builds fine locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1775699568
Sphinxes often speak in riddles, this is something I often repeat (too often I am afraid)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1812510053
Ho about completing it :)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR closed pull request #34809: Migrate provider-specific executor docs to providers
URL: https://github.com/apache/airflow/pull/34809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1773825770
https://github.com/apache/airflow/pull/35102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1773846644
Rebased. Let's see if the fix works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1775621186
My wild guess is that you have label reference somewhere in the code left (of the form `_executor:celeryexecutor`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1773877109
Thanks. What's the best way for me to retry the failed test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1848203799
@lzdanski This is the spell check sphinx issue I was referring to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1765555394
Seems there are conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Migrate provider-specific executor docs to providers [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34809:
URL: https://github.com/apache/airflow/pull/34809#issuecomment-1906994214
Fix to failing release Helm job - https://github.com/apache/airflow/pull/36985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org