You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "o-nikolas (via GitHub)" <gi...@apache.org> on 2023/09/13 00:34:01 UTC
[GitHub] [airflow] o-nikolas opened a new pull request, #34324: Executors doc update
o-nikolas opened a new pull request, #34324:
URL: https://github.com/apache/airflow/pull/34324
Added new sections describing the BaseExecutor interface in more detail to help users in writing custom executors.
This addresses the last AIP-51 task (#27934).
A screenshot of the fully rendered page can be viewed below:

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #34324: Executors doc update
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1718299739
Thanks for the feedback @potiuk, replies inline below :)
> Small comment - not directly this but related and clearly visible in the rendered page. Can you please move the "Testing" documentation out of the executor hierarchy ? The two Howtos seems completely out of place for the document.
I don't see a "Testing" doc in the executor hierarchy? I'm likely missing something though :eyes:
>
> I like the document. The few proposals I would like to propose:
>
> * Instead of "strongly recommend" I would write "Sequential Executor is not suitable for production. It does not allow for parallel task runnning and due to that, some Airflow features (for example running sensors waiting for outputs of another task are not working properly).
This section actually already existed before I updated it (my changes start at "Writing Your Own Executor"), but I can make this change anyway while I'm here.
> * Maybe it would be possible to Have a boilerplate template of an executor as part of the docs ? That would probably significantly decrease the friction of writing new executor (on the other hand, maybe we actually want some friction). Writing your own executor is not an easy / simple task that everyone should even attempt
I did think of this, but unlike custom operators, it's quite a large piece of code to even have a boilerplate. I was thinking of adding a very pseudo code snippet, but wasn't convinced that it would be helpful beyond folks just having a look at any of the existing community executors which all implement the BaseExecutor interface. Interested to hear what others think!
> * Speaking of which - I think we should explaine why somoene would like to write their own executor. Short paragraph describing a decision making process here and some example reasons why one would rather write their own executor than choose from the existing ones, would be great.
Great suggestion. I'll add a section and come up with a list of bullets for why you'd want to create a custom executor. If you have any to add, please reply with them here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #34324: Executors doc update
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1737976416
@syedahsn
I added some pseudo code to the CLI and logging sections to give a more concrete idea of how to use those parts of the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #34324: Executors doc update
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1736099421
> Looks really good. Very well done! Just a comment, and I'm not sure how much can be done about this, but the Logging and CLI sections don't really add a lot of information about _how_ to log or vend CLI commands. I don't know what the process for that is, and whether that's something that should be discussed in a document like this. But if a bit more information (or links to examples from the LocalExecutor or Kubernetes Executor) can be added about that, it would help I think.
Thanks for having a look @syedahsn
I can try add a bit more info. But so far I've just mentioned which executors make use of it, rather than linking directly to code, since that link will very quickly become stale as changes are made to that class. Maybe I can put some pseudo code in here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #34324: Executors doc update
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1719999297
@potiuk
Changes are in! It was quite a sticky mess with the debug executor doc. The fix I landed on was just shuffling around the headings on that page to not have 3 title level headings (which actually makes sense, I think that was a bug before). And move the DebugExecutor section to the top (which makes sense, since it _is_ the DebugExecutor doc page, the other info is additional).
Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #34324: Executors doc update
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1716781521
CC @eladkal @potiuk and @Bowrna who showed interested in this task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executors doc update [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1756036373
This has been open for a while and the feedback has stopped trickling in. @potiuk, @vincbeck, @syedahsn, @ferruzzi I think I've addressed all your feedback. Merging this one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Executors doc update [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas merged PR #34324:
URL: https://github.com/apache/airflow/pull/34324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #34324: Executors doc update
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1718323312
> I don't see a "Testing" doc in the executor hierarchy? I'm likely missing something though 👀
<img width="851" alt="Screenshot 2023-09-13 at 23 06 11" src="https://github.com/apache/airflow/assets/595491/acdacc51-415d-4296-af58-89764c324282">
What I see:
* Testing ....
* Debugging ....
* DebugExecutor (Deprecated)
I think it's because those files are wrongly placed :)
> This section actually already existed before I updated it (my changes start at "Writing Your Own Executor"), but I can make this change anyway while I'm here.
Yep. It's all "Let's make the doc complete and makes sensee" rather than "Let's preserve what was wrong before"
> I did think of this, but unlike custom operators, it's quite a large piece of code to even have a boilerplate. I was thinking of adding a very pseudo code snippet, but wasn't convinced that it would be helpful beyond folks just having a look at any of the existing community executors which all implement the BaseExecutor interface. Interested to hear what others think!
Agree. It should be enough to say ("look at existing executor's implementation to take inspiration" - which I think is already there, so unless there is some nicer way to surface, it's more of a "nice to have".
> I'll add a section and come up with a list of bullets for why you'd want to create a custom executor. If you have any to add, please reply with them here!
Will do !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on a diff in pull request #34324: Executors doc update
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #34324:
URL: https://github.com/apache/airflow/pull/34324#discussion_r1339125938
##########
docs/apache-airflow/core-concepts/executor/debug.rst:
##########
@@ -15,8 +15,47 @@
specific language governing permissions and limitations
under the License.
+.. _executor:DebugExecutor:
Review Comment:
No changes here other than bringing this section to the top and modifying section hierarchy to satisfy the static checks we have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #34324: Executors doc update
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #34324:
URL: https://github.com/apache/airflow/pull/34324#discussion_r1339169842
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
Review Comment:
Might just be me, but the colon at the end here feels awkward. It is followed by a note block, then by another statement which ends in a colon.
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
+
+.. note::
+ For more information about Airflow's public interface see :doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service but from your preferred cloud provider.
+* You have a private tool/service for task execution that is only available to you or your organization.
+
+
+Important BaseExecutor Methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These methods don't require overriding to implement your own executor, but are useful to be aware of:
+
+* ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks.
+* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor.
+* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing.
+* ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running.
+* ``send_callback``: Sends any callbacks to the sink configured on the executor.
+
+
+Mandatory Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods must be overridden at minimum to have your executor supported by Airflow:
+
+* ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler.
+* ``execute_async``: executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
+
+
+Optional Interface Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods aren't required to override to have a functional Airflow executor. However, some powerful capabilities and stability can come from implementing them:
+
+* ``start``: The Airflow scheduler (and backfill) job will call this method after it initializes the executor object. Any additional setup required by the executor can be completed here.
+* end: The Airflow scheduler (and backfill) job will call this method as it is tearing down. Any synchronous cleanup required to finish running jobs should be done here.
Review Comment:
```suggestion
* ``end``: The Airflow scheduler (and backfill) job will call this method as it is tearing down. Any synchronous cleanup required to finish running jobs should be done here.
```
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
+
+.. note::
+ For more information about Airflow's public interface see :doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service but from your preferred cloud provider.
+* You have a private tool/service for task execution that is only available to you or your organization.
+
+
+Important BaseExecutor Methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These methods don't require overriding to implement your own executor, but are useful to be aware of:
+
+* ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks.
+* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor.
+* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing.
+* ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running.
+* ``send_callback``: Sends any callbacks to the sink configured on the executor.
+
+
+Mandatory Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods must be overridden at minimum to have your executor supported by Airflow:
+
+* ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler.
+* ``execute_async``: executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
+
+
+Optional Interface Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods aren't required to override to have a functional Airflow executor. However, some powerful capabilities and stability can come from implementing them:
+
+* ``start``: The Airflow scheduler (and backfill) job will call this method after it initializes the executor object. Any additional setup required by the executor can be completed here.
+* end: The Airflow scheduler (and backfill) job will call this method as it is tearing down. Any synchronous cleanup required to finish running jobs should be done here.
+* ``terminate``: More forcefully stop the executor, even killing/stopping in flight tasks instead of synchronously waiting for completion.
Review Comment:
Not sure about this one, up to you.
```suggestion
* ``terminate``: More forcefully stop the executor, even killing/stopping in-flight tasks instead of synchronously waiting for completion.
```
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
+
+.. note::
+ For more information about Airflow's public interface see :doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service but from your preferred cloud provider.
+* You have a private tool/service for task execution that is only available to you or your organization.
+
+
+Important BaseExecutor Methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These methods don't require overriding to implement your own executor, but are useful to be aware of:
+
+* ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks.
+* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor.
+* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing.
+* ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running.
+* ``send_callback``: Sends any callbacks to the sink configured on the executor.
+
+
+Mandatory Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods must be overridden at minimum to have your executor supported by Airflow:
+
+* ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler.
+* ``execute_async``: executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
+
+
+Optional Interface Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods aren't required to override to have a functional Airflow executor. However, some powerful capabilities and stability can come from implementing them:
+
+* ``start``: The Airflow scheduler (and backfill) job will call this method after it initializes the executor object. Any additional setup required by the executor can be completed here.
+* end: The Airflow scheduler (and backfill) job will call this method as it is tearing down. Any synchronous cleanup required to finish running jobs should be done here.
+* ``terminate``: More forcefully stop the executor, even killing/stopping in flight tasks instead of synchronously waiting for completion.
+* ``cleanup_stuck_queued_tasks``: If tasks are stuck in the queued state for longer than ``task_queued_timeout`` then they are collected by the scheduler and provided to the executor to have an opportunity to handle them (perform any graceful cleanup/teardown) via this method and return the Task Instances for a warning message displayed to users.
+* ``try_adopt_task_instances``: Tasks that have been abandoned (e.g. from a scheduler job that died) are provided to the executor to adopt or otherwise handle them via this method. Any tasks that cannot be adopted (by default the BaseExector assumes all cannot be adopted) should be returned.
+* ``get_cli_commands``: Executors may vend CLI commands to users by implementing this method, see the `CLI`_ section below for more details.
+* ``get_task_log``: Executors may vend log messages to Airflow task logs by implementing this method, see the `Logging`_ section below for more details.
+
+Compatibility Attributes
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``BaseExecutor`` class interface contains a set of attributes that Airflow core code uses to check the features that your executor is compatible with. When writing your own Airflow executor be sure to set these correctly for your use case. Each attribute is simply a boolean to enable/disable a feature or indicate that a feature is supported/unsupported by the executor:
+
+* ``supports_pickling``: Whether or not the executor supports reading pickled DAGs from the Database before execution (rather than reading the DAG definition from the file system).
+* ``supports_sentry``: Whether or not the executor supports `Sentry <https://sentry.io>`_.
+
+* ``is_local``: Whether or not the executor is remote or local. See the `Executor Types`_ section above.
+* ``is_single_threaded``: Whether or not the executor is single threaded. This is particularly relevant to what database backends are supported. Single threaded executors can run with any backend, including SQLite.
+* ``is_production``: Whether or not the executor should be used for production purposes. A UI message is displayed to users when they are using a non-production ready executor.
+
+* ``change_sensor_mode_to_reschedule``: Running Airflow sensors in poke mode can block the thread of executors and in some cases Airflow.
+* ``serve_logs``: Whether or not the executor supports serving logs, see :doc:`/administration-and-deployment/logging-monitoring/logging-tasks`.
+
+CLI
+^^^
+
+Executors may vend CLI commands which will be included in the ``airflow`` command line tool by implementing the ``get_cli_commands`` method. Executors such as ``CeleryExecutor`` and ``KubernetesExecutor`` for example, make use of this mechanism. The commands can be used to setup required workers, initialize environment or set other configuration. Commands are only vended for the currently configured executor. A pseudo-code example of implementing CLI command vending from an executor can be seen below:
+
+.. code-block:: python
+
+ @staticmethod
+ def get_cli_commands() -> list[GroupCommand]:
+ sub_commands = [
+ ActionCommand(
+ name="command_name",
+ help="Description of what this specific command does",
+ func=lazy_load_command("path.to.python.function.for.command"),
+ args=(),
+ ),
+ ]
+
+ return [
+ GroupCommand(
+ name="my_cool_executor",
+ help="Description of what this group of commands do",
+ subcommands=sub_commands,
+ ),
+ ]
+
+.. note::
+ There are no strict rules in place, currently, for the Airflow command namespace. It is up to developers to use names for their CLI commands that are sufficiently unique so as to not cause conflicts with other Airflow executors or components.
Review Comment:
```suggestion
Currently there are no strict rules in place for the Airflow command namespace. It is up to developers to use names for their CLI commands that are sufficiently unique so as to not cause conflicts with other Airflow executors or components.
```
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
+
+.. note::
+ For more information about Airflow's public interface see :doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service but from your preferred cloud provider.
+* You have a private tool/service for task execution that is only available to you or your organization.
+
+
+Important BaseExecutor Methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These methods don't require overriding to implement your own executor, but are useful to be aware of:
+
+* ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks.
+* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor.
+* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing.
+* ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running.
+* ``send_callback``: Sends any callbacks to the sink configured on the executor.
+
+
+Mandatory Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods must be overridden at minimum to have your executor supported by Airflow:
+
+* ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler.
+* ``execute_async``: executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
Review Comment:
```suggestion
* ``execute_async``: Executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on a diff in pull request #34324: Executors doc update
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #34324:
URL: https://github.com/apache/airflow/pull/34324#discussion_r1339165472
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
+
+.. note::
+ For more information about Airflow's public interface see :doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service but from your preferred cloud provider.
+* You have a private tool/service for task execution that is only available to you or your organization.
+
+
+Important BaseExecutor Methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These methods don't require overriding to implement your own executor, but are useful to be aware of:
+
+* ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks.
+* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor.
+* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing.
+* ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running.
+* ``send_callback``: Sends any callbacks to the sink configured on the executor.
+
+
+Mandatory Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods must be overridden at minimum to have your executor supported by Airflow:
+
+* ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler.
+* ``execute_async``: executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
+
+
+Optional Interface Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods aren't required to override to have a functional Airflow executor. However, some powerful capabilities and stability can come from implementing them:
+
+* ``start``: The Airflow scheduler (and backfill) job will call this method after it initializes the executor object. Any additional setup required by the executor can be completed here.
+* end: The Airflow scheduler (and backfill) job will call this method as it is tearing down. Any synchronous cleanup required to finish running jobs should be done here.
Review Comment:
```suggestion
* ``end``: The Airflow scheduler (and backfill) job will call this method as it is tearing down. Any synchronous cleanup required to finish running jobs should be done here.
```
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
+
+.. note::
+ For more information about Airflow's public interface see :doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service but from your preferred cloud provider.
Review Comment:
Is the "but" necessary here?
```suggestion
* You'd like to use an executor that leverages a compute service from your preferred cloud provider.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #34324: Executors doc update
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #34324:
URL: https://github.com/apache/airflow/pull/34324#discussion_r1339321927
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
+
+.. note::
+ For more information about Airflow's public interface see :doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service but from your preferred cloud provider.
+* You have a private tool/service for task execution that is only available to you or your organization.
+
+
+Important BaseExecutor Methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These methods don't require overriding to implement your own executor, but are useful to be aware of:
+
+* ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks.
+* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor.
+* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing.
+* ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running.
+* ``send_callback``: Sends any callbacks to the sink configured on the executor.
+
+
+Mandatory Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods must be overridden at minimum to have your executor supported by Airflow:
+
+* ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler.
+* ``execute_async``: executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
+
+
+Optional Interface Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods aren't required to override to have a functional Airflow executor. However, some powerful capabilities and stability can come from implementing them:
+
+* ``start``: The Airflow scheduler (and backfill) job will call this method after it initializes the executor object. Any additional setup required by the executor can be completed here.
+* end: The Airflow scheduler (and backfill) job will call this method as it is tearing down. Any synchronous cleanup required to finish running jobs should be done here.
+* ``terminate``: More forcefully stop the executor, even killing/stopping in flight tasks instead of synchronously waiting for completion.
+* ``cleanup_stuck_queued_tasks``: If tasks are stuck in the queued state for longer than ``task_queued_timeout`` then they are collected by the scheduler and provided to the executor to have an opportunity to handle them (perform any graceful cleanup/teardown) via this method and return the Task Instances for a warning message displayed to users.
+* ``try_adopt_task_instances``: Tasks that have been abandoned (e.g. from a scheduler job that died) are provided to the executor to adopt or otherwise handle them via this method. Any tasks that cannot be adopted (by default the BaseExector assumes all cannot be adopted) should be returned.
+* ``get_cli_commands``: Executors may vend CLI commands to users by implementing this method, see the `CLI`_ section below for more details.
+* ``get_task_log``: Executors may vend log messages to Airflow task logs by implementing this method, see the `Logging`_ section below for more details.
+
+Compatibility Attributes
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``BaseExecutor`` class interface contains a set of attributes that Airflow core code uses to check the features that your executor is compatible with. When writing your own Airflow executor be sure to set these correctly for your use case. Each attribute is simply a boolean to enable/disable a feature or indicate that a feature is supported/unsupported by the executor:
+
+* ``supports_pickling``: Whether or not the executor supports reading pickled DAGs from the Database before execution (rather than reading the DAG definition from the file system).
+* ``supports_sentry``: Whether or not the executor supports `Sentry <https://sentry.io>`_.
+
+* ``is_local``: Whether or not the executor is remote or local. See the `Executor Types`_ section above.
+* ``is_single_threaded``: Whether or not the executor is single threaded. This is particularly relevant to what database backends are supported. Single threaded executors can run with any backend, including SQLite.
+* ``is_production``: Whether or not the executor should be used for production purposes. A UI message is displayed to users when they are using a non-production ready executor.
+
+* ``change_sensor_mode_to_reschedule``: Running Airflow sensors in poke mode can block the thread of executors and in some cases Airflow.
+* ``serve_logs``: Whether or not the executor supports serving logs, see :doc:`/administration-and-deployment/logging-monitoring/logging-tasks`.
+
+CLI
+^^^
+
+Executors may vend CLI commands which will be included in the ``airflow`` command line tool by implementing the ``get_cli_commands`` method. Executors such as ``CeleryExecutor`` and ``KubernetesExecutor`` for example, make use of this mechanism. The commands can be used to setup required workers, initialize environment or set other configuration. Commands are only vended for the currently configured executor. A pseudo-code example of implementing CLI command vending from an executor can be seen below:
+
+.. code-block:: python
+
+ @staticmethod
+ def get_cli_commands() -> list[GroupCommand]:
+ sub_commands = [
+ ActionCommand(
+ name="command_name",
+ help="Description of what this specific command does",
+ func=lazy_load_command("path.to.python.function.for.command"),
+ args=(),
+ ),
+ ]
+
+ return [
+ GroupCommand(
+ name="my_cool_executor",
+ help="Description of what this group of commands do",
+ subcommands=sub_commands,
+ ),
+ ]
+
+.. note::
+ There are no strict rules in place, currently, for the Airflow command namespace. It is up to developers to use names for their CLI commands that are sufficiently unique so as to not cause conflicts with other Airflow executors or components.
Review Comment:
As a comma-abuser, I feel bad about this suggestion, but I do think it flows a bit better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #34324: Executors doc update
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1716978118
Small comment - not directly this but related and clearly visible in the rendered page. Can you please move the "Testing" documentation out of the executor hierarchy ? The two Howtos seems completely out of place for the document.
I like the document. The few proposals I would like to propose:
* Instead of "strongly recommend" I would write "Sequential Executor is not suitable for production. It does not allow for parallel task runnning and due to that, some Airflow features (for example running sensors waiting for outputs of another task are not working properly).
* Maybe it would be possible to Have a boilerplate template of an executor as part of the docs ? That would probably significantly decrease the friction of writing new executor (on the other hand, maybe we actually want some friction). Writing your own executor is not an easy / simple task that everyone should even attempt
* Speaking of which - I think we should explaine why somoene would like to write their own executor. Short paragraph describing a decision making process here and some example reasons why one would rather write their own executor than choose from the existing ones, would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] syedahsn commented on pull request #34324: Executors doc update
Posted by "syedahsn (via GitHub)" <gi...@apache.org>.
syedahsn commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1734684475
Looks really good. Very well done! Just a comment, and I'm not sure how much can be done about this, but the Logging and CLI sections don't really add a lot of information about _how_ to log or vend CLI commands. I don't know what the process for that is, and whether that's something that should be discussed in a document like this. But if a bit more information (or links to examples from the LocalExecutor or Kubernetes Executor) can be added about that, it would help I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on a diff in pull request #34324: Executors doc update
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #34324:
URL: https://github.com/apache/airflow/pull/34324#discussion_r1339125938
##########
docs/apache-airflow/core-concepts/executor/debug.rst:
##########
@@ -15,8 +15,47 @@
specific language governing permissions and limitations
under the License.
+.. _executor:DebugExecutor:
Review Comment:
No changes here other than bringing this section to the top to satisfy the static checks we have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on pull request #34324: Executors doc update
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #34324:
URL: https://github.com/apache/airflow/pull/34324#issuecomment-1718345683
> > I don't see a "Testing" doc in the executor hierarchy? I'm likely missing something though 👀
>
> <img alt="Screenshot 2023-09-13 at 23 06 11" width="851" src="https://user-images.githubusercontent.com/595491/267789481-acdacc51-415d-4296-af58-89764c324282.png">
>
> What I see:
>
> * Testing ....
>
> * Debugging ....
>
> * DebugExecutor (Deprecated)
>
>
> I think it's because those files are wrongly placed :)
Ah! In the types subsection. I was looking for it in the table of contents hierarchy on the left.
Those three sections all come from the DebugExecutor doc. Which has been modified to move the section on the DebugExecutor itself to the bottom of the page and new sections were added for testing dags without the debug executor (now that it's deprecated), and some comparisons with the debug executor itself. It's a little tricky now how to handle that with the `.. toctree::` directive we use for those lists.
The easiest thing may be to just remove the debug executor from the `.. toctree::` directive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on a diff in pull request #34324: Executors doc update
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #34324:
URL: https://github.com/apache/airflow/pull/34324#discussion_r1339320655
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
Review Comment:
Yupp, fair point. I'll just terminate that in a period!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #34324: Executors doc update
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #34324:
URL: https://github.com/apache/airflow/pull/34324#discussion_r1339180017
##########
docs/apache-airflow/core-concepts/executor/index.rst:
##########
@@ -78,3 +71,134 @@ There are two types of executor - those that run tasks *locally* (inside the ``s
.. note::
New Airflow users may assume they need to run a separate executor process using one of the Local or Remote Executors. This is not correct. The executor logic runs *inside* the scheduler process, and will run the tasks locally or not depending the executor selected.
+
+Writing Your Own Executor
+-------------------------
+
+All Airflow executors implement a common interface so that they are pluggable and any executor has access to all abilities and integrations within Airflow. Primarily, the Airflow scheduler uses this interface to interact with the executor, but other components such as logging, CLI and backfill do as well.
+The public interface is the :class:`~airflow.executors.base_executor.BaseExecutor`. You can look through the code for the most detailed and up to date interface, but some important highlights are outlined below:
+
+.. note::
+ For more information about Airflow's public interface see :doc:`/public-airflow-interface`.
+
+Some reasons you may want to write a custom executor include:
+
+* An executor does not exist which fits your specific use case, such as a specific tool or service for compute.
+* You'd like to use an executor that leverages a compute service but from your preferred cloud provider.
+* You have a private tool/service for task execution that is only available to you or your organization.
+
+
+Important BaseExecutor Methods
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These methods don't require overriding to implement your own executor, but are useful to be aware of:
+
+* ``heartbeat``: The Airflow scheduler Job loop will periodically call heartbeat on the executor. This is one of the main points of interaction between the Airflow scheduler and the executor. This method updates some metrics, triggers newly queued tasks to execute and updates state of running/completed tasks.
+* ``queue_command``: The Airflow Executor will call this method of the BaseExecutor to provide tasks to be run by the executor. The BaseExecutor simply adds the TaskInstances to an internal list of queued tasks within the executor.
+* ``get_event_buffer``: The Airflow scheduler calls this method to retrieve the current state of the TaskInstances the executor is executing.
+* ``has_task``: The scheduler uses this BaseExecutor method to determine if an executor already has a specific task instance queued or running.
+* ``send_callback``: Sends any callbacks to the sink configured on the executor.
+
+
+Mandatory Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods must be overridden at minimum to have your executor supported by Airflow:
+
+* ``sync``: Sync will get called periodically during executor heartbeats. Implement this method to update the state of the tasks which the executor knows about. Optionally, attempting to execute queued tasks that have been received from the scheduler.
+* ``execute_async``: executes a command asynchronously. A command in this context is an Airflow CLI command to run an Airflow task. This method is called (after a few layers) during executor heartbeat which is run periodically by the scheduler. In practice, this method often just enqueues tasks into an internal or external queue of tasks to be run (e.g. ``KubernetesExecutor``). But can also execute the tasks directly as well (e.g. ``LocalExecutor``). This will depend on the executor.
+
+
+Optional Interface Methods to Implement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following methods aren't required to override to have a functional Airflow executor. However, some powerful capabilities and stability can come from implementing them:
+
+* ``start``: The Airflow scheduler (and backfill) job will call this method after it initializes the executor object. Any additional setup required by the executor can be completed here.
+* end: The Airflow scheduler (and backfill) job will call this method as it is tearing down. Any synchronous cleanup required to finish running jobs should be done here.
Review Comment:
Vincent pointed this one out while I was typing. I'll resolve mine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org