You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by js...@apache.org on 2025/09/27 16:17:50 UTC
(airflow) branch main updated: Revise details of edge documentation after Airflow 3.1 release (#56166)
This is an automated email from the ASF dual-hosted git repository.
jscheffl pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new d4a6a998b81 Revise details of edge documentation after Airflow 3.1 release (#56166)
d4a6a998b81 is described below
commit d4a6a998b81b7229142ea6b3005dae18f9be45df
Author: Jens Scheffler <95...@users.noreply.github.com>
AuthorDate: Sat Sep 27 18:17:43 2025 +0200
Revise details of edge documentation after Airflow 3.1 release (#56166)
---
providers/edge3/docs/architecture.rst | 21 ++++++++++-----------
providers/edge3/docs/deployment.rst | 26 ++++++++++++++++++++------
providers/edge3/docs/edge_executor.rst | 13 +++++++++----
providers/edge3/docs/ui_plugin.rst | 6 +++---
4 files changed, 42 insertions(+), 24 deletions(-)
diff --git a/providers/edge3/docs/architecture.rst b/providers/edge3/docs/architecture.rst
index 8ea8959e70d..5c38b1ab95d 100644
--- a/providers/edge3/docs/architecture.rst
+++ b/providers/edge3/docs/architecture.rst
@@ -60,10 +60,10 @@ deployed outside of the central Airflow cluster is connected via HTTP(s) to the
edge_worker->api[label="HTTP(s)"]
}
-* **Workers** - Execute the assigned tasks - most standard setup has local or centralized workers, e.g. via Celery
+* **Workers** (Central) - Execute the assigned tasks - most standard setup has local or centralized workers, e.g. via Celery
* **Edge Workers** - Special workers which pull tasks via HTTP(s) as provided as feature via this provider package
* **Scheduler** - Responsible for adding the necessary tasks to the queue. The EdgeExecutor is running as a module inside the scheduler.
-* **API server** - HTTP REST API Server provides access to Dag/task status information. The required end-points are
+* **API server** (webserver in Airflow 2.x) - HTTP REST API Server provides access to Dag/task status information. The required end-points are
provided by the Edge provider plugin. The Edge Worker uses this API to pull tasks and send back the results.
* **Database** - Contains information about the status of tasks, Dags, Variables, connections, etc.
@@ -73,19 +73,18 @@ In detail the parts of the Edge provider are deployed as follows:
:alt: Overview and communication of Edge Provider modules
* **EdgeExecutor** - The EdgeExecutor is running inside the core Airflow scheduler. It is responsible for
- scheduling tasks and sending them to the Edge job queue in the database. The EdgeExecutor is a subclass of the
- ``airflow.executors.base_executor.BaseExecutor`` class. To activate the EdgeExecutor, you need to set the
- ``executor`` configuration option in the ``airflow.cfg`` file to
+ scheduling tasks and sending them to the Edge job queue in the database. To activate the EdgeExecutor, you
+ need to set the ``executor`` configuration option in the ``airflow.cfg`` file to
``airflow.providers.edge3.executors.EdgeExecutor``. For more details see :doc:`edge_executor`. Note that also
multiple executors can be used in parallel together with the EdgeExecutor.
-* **API server** - The API server is providing REST endpoints to the web UI as well as serves static files. The
- Edge provider adds a plugin that provides additional REST API for the Edge Worker as well as UI elements to
- manage workers (currently Airflow 2.10 only).
+* **API server** (webserver in Airflow 2.x) - The API server is providing REST endpoints to the web UI as well
+ as serves static files. The Edge provider adds a plugin that provides additional REST API for the Edge Worker
+ as well as UI elements to manage workers (not available in Airflow 3.0).
The API server is responsible for handling requests from the Edge Worker and sending back the results. To
activate the API server, you need to set the ``api_enabled`` configuration option in ``edge`` section in the
``airflow.cfg`` file to ``True``. The API endpoints for edge is not started by default.
- Fr more details see :doc:`ui_plugin`.
-* **Database** - The Airflow meta database is used to store the status of tasks, Dags, Variables, connections
+ For more details see :doc:`ui_plugin`.
+* **Database** - The Airflow meta database is used to store the status of tasks, Dags, Variables, Connections
etc. The Edge provider uses the database to store the status of the Edge Worker instances and the tasks that
are assigned to it. The database is also used to store the results of the tasks that are executed by the
Edge Worker. Setup of needed tables and migration is done automatically when the provider package is deployed.
@@ -174,7 +173,7 @@ The following features are known missing and will be implemented in increments:
- Test/Support on Windows for Edge Worker
- Scaling test - Check and define boundaries of workers/jobs. Today it is known to
- scale into a range of 50 workers. This is not a hard limit but just an experience reported.
+ scale into a range of ~80 workers. This is not a hard limit but just an experience reported.
- Load tests - impact of scaled execution and code optimization
- Incremental logs during task execution can be served w/o shared log disk on api-server
- Reduce dependencies during execution: Today the worker depends on the airflow core with a lot
diff --git a/providers/edge3/docs/deployment.rst b/providers/edge3/docs/deployment.rst
index 9aa741290a2..0cfe2fa4d98 100644
--- a/providers/edge3/docs/deployment.rst
+++ b/providers/edge3/docs/deployment.rst
@@ -21,8 +21,7 @@ Edge Worker Deployment
Edge Workers can be deployed outside of the central Airflow infrastructure. They
are connected to the Airflow API server via HTTP(s). The Edge Worker is a
lightweight component that can be deployed on any machine that has outbound
-HTTP(s) access to the Airflow API server. The Edge Worker is designed to be
-lightweight and easy to deploy. It allows you to run Airflow tasks on machines
+HTTP(s) access to the Airflow API server. It allows you to run Airflow tasks on machines
that are not part of your main data center, e.g. edge servers. This also allows to
deploy only reduced dependencies on the edge worker.
@@ -46,7 +45,8 @@ Minimum Airflow configuration settings for the Edge Worker to make it running is
- Section ``[api_auth]``
- - ``jwt_secret``: A matching secret to that on the api-server (starting from version 3.0.0).
+ - ``jwt_secret``: A shared secret between workers and the api-server to authenticate
+ (starting from version 3.0.0).
- Section ``[core]``
@@ -56,7 +56,7 @@ Minimum Airflow configuration settings for the Edge Worker to make it running is
- ``executor``: Executor must be set or added to be ``airflow.providers.edge3.executors.EdgeExecutor``
- ``internal_api_secret_key``: An encryption key must be set on api-server and Edge Worker component as
shared secret to authenticate traffic. It should be a random string like the fernet key
- (for versions earlier than 3.0.0).
+ (for versions earlier than 3.0.0 - from Airflow 3.0 and above this is using ``api_auth.jwt_secret``).
- Section ``[edge]``
@@ -72,6 +72,20 @@ subcommand
.. code-block:: bash
airflow edge worker
+ 2025-09-27T12:28:32.954316Z [info ] Starting worker with API endpoint http://localhost:8080/edge_worker/v1/rpcapi
+ ____________ _____________
+ ____ |__( )_________ __/__ /________ __
+ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
+ ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
+ _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
+ ____ __ _ __ __
+ / __/__/ /__ ____ | | /| / /__ ____/ /_____ ____
+ / _// _ / _ / -_) | |/ |/ / _ \/ __/ _/ -_) __/
+ /___/\_,_/\_, /\__/ |__/|__/\___/_/ /_/\_\\__/_/
+ /___/
+
+ 2025-09-27T12:28:33.171525Z [info ] No new job to process
+
You can also start this worker in the background by running
it as a daemonized process. Additionally, you can redirect stdout
@@ -97,9 +111,9 @@ process as and wait until all running tasks are completed. Also in a console you
``Ctrl-C`` to stop the worker.
If you want to monitor the remote activity and worker, use the UI plugin which
-is included in the provider package and install it on the webserver and use the
+is included in the provider package and install it on the api-server / webserver and use the
"Admin" - "Edge Worker Hosts" and "Edge Worker Jobs" pages.
-(Note: The plugin is not ported to Airflow 3.0 web UI at time of writing)
+(Note: The plugin is not available on Airflow 3.0 UI, it is only in 2.x and 3.1++)
If you want to check status of the worker via CLI you can use the command
diff --git a/providers/edge3/docs/edge_executor.rst b/providers/edge3/docs/edge_executor.rst
index 3d6c7644eb6..c8e8fa9abec 100644
--- a/providers/edge3/docs/edge_executor.rst
+++ b/providers/edge3/docs/edge_executor.rst
@@ -27,6 +27,9 @@ The configuration parameters of the Edge Executor can be found in the Edge provi
To understand the setup of the Edge Executor, please also take a look to :doc:`architecture`.
+ See more details Airflow documentation
+ :ref:`apache-airflow:using-multiple-executors-concurrently`.
+
.. _edge_executor:queue:
Queues
@@ -43,6 +46,7 @@ Workers can listen to one or multiple queues of tasks. When a worker is
started (using command ``airflow edge worker``), a set of comma-delimited queue
names (with no whitespace) can be given (e.g. ``airflow edge worker -q remote,wisconsin_site``).
This worker will then only pick up tasks wired to the specified queue(s).
+If the ``queue`` attribute is not given then a worker will pick tasks from all queues.
This can be useful if you need specialized workers, either from a
resource perspective (for say very lightweight tasks where one worker
@@ -104,9 +108,10 @@ Current Limitations Edge Executor
- Some known limitations
- - Log upload will only work if you use a single web server instance or they need to share one log file volume.
- Logs are uploaded in chunks and are transferred via API. If you use multiple webservers w/o a shared log volume
- the logs will be scattered across the webserver instances.
+ - Log upload will only work if you use a single api-server / webserver instance or they need to share one log file
+ volume. Logs are uploaded in chunks and are transferred via API. If you use multiple api-servers / webservers w/o
+ a shared log volume the logs will be scattered across the api-server / webserver instances and if you view the
+ logs on UI you will only see fractions of the logs.
- Performance: No extensive performance assessment and scaling tests have been made. The edge executor package is
optimized for stability. This will be incrementally improved in future releases. Setups have reported stable
- operation with ~80 workers until now. Note that executed tasks require more webserver API capacity.
+ operation with ~80 workers until now. Note that executed tasks require more api-server / webserver API capacity.
diff --git a/providers/edge3/docs/ui_plugin.rst b/providers/edge3/docs/ui_plugin.rst
index 1f639c81458..043892e1ffc 100644
--- a/providers/edge3/docs/ui_plugin.rst
+++ b/providers/edge3/docs/ui_plugin.rst
@@ -22,7 +22,7 @@ The Edge provider uses a Plugin to
- Extend the REST API endpoints for connecting workers to the Airflow cluster
- Provide a web UI for managing the workers and monitoring their status and tasks
- (Note: Airflow 3.0 does not have support for UI plugins. The UI plugin is only available in Airflow 2.10 and in 3.1 and newer.)
+ (Note: Airflow 3.0 does not have support for UI plugins. The UI plugin is only available in Airflow 2.x and in 3.1 and newer.)
REST API endpoints
------------------
@@ -35,14 +35,14 @@ The Edge provider adds the following REST API endpoints to the Airflow API:
- ``/edge_worker/v1/health``: Check that the API endpoint is deployed and active
To see full documentation of the API endpoints open the Airflow web UI and navigate to
-the sub-path ``/edge_worker/docs`` (Airflow 3.0) or ``/edge_worker/v1/ui`` (Airflow 2.10).
+the sub-path ``/edge_worker/docs`` (Airflow 3.0) or ``/edge_worker/v1/ui`` (Airflow 2.x).
Web UI Plugin
-------------
.. note::
- Airflow 3.0 does not support UI plugins. The UI plugin is only available in Airflow 2.10 and in 3.1 and newer.
+ Airflow 3.0 does not support UI plugins. The UI plugin is only available in Airflow 2.x and in 3.1 and newer.
Alternatively you can use the CLI commands as described in :ref:`deployment:maintenance-mgmt-cli`.
The Edge provider adds a web UI plugin to the Airflow web UI. The plugin is