You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ep...@apache.org on 2025/12/09 17:19:43 UTC

(airflow) 04/32: [v3-1-test] Update the documentation for the LocalExecutor (#58990) (#59022)

This is an automated email from the ASF dual-hosted git repository.

ephraimanierobi pushed a commit to branch v3-1-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 833911fe19dbbe1995a71995b12fb705c4a310dc
Author: github-actions[bot] <41...@users.noreply.github.com>
AuthorDate: Thu Dec 4 02:11:53 2025 +0100

    [v3-1-test] Update the documentation for the LocalExecutor (#58990) (#59022)
    
    (cherry picked from commit 729571b1a8d6009b86fd24d8482907894da8a4f1)
    
    Co-authored-by: Jeongwoo Do <48...@users.noreply.github.com>
---
 airflow-core/docs/core-concepts/executor/local.rst | 23 ++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/airflow-core/docs/core-concepts/executor/local.rst b/airflow-core/docs/core-concepts/executor/local.rst
index a533ae3a5b8..dc7a70fd873 100644
--- a/airflow-core/docs/core-concepts/executor/local.rst
+++ b/airflow-core/docs/core-concepts/executor/local.rst
@@ -29,8 +29,27 @@ This parameter must be greater than ``0``.
 The :class:`~airflow.executors.local_executor.LocalExecutor` spawns the number of processes equal to the value of ``self.parallelism`` at
 ``start`` time, using a ``task_queue`` to coordinate the ingestion of tasks and the work distribution among the workers, which will take
 a task as soon as they are ready. During the lifecycle of the LocalExecutor, the worker processes are running waiting for tasks, once the
-LocalExecutor receives the call to shutdown the executor a poison token is sent to the workers to terminate them. Processes used in this
-strategy are of class :class:`~airflow.executors.local_executor.QueuedLocalWorker`.
+LocalExecutor receives the call to shutdown the executor a poison token is sent to the workers to terminate them.
+
+The worker spawning behavior differs based on the multiprocessing start method:
+
+- **Fork mode** (default on Linux): Workers are spawned all at once up to ``parallelism`` to prevent memory spikes
+  caused by Copy-on-Write (COW). See `Discussion <https://github.com/apache/airflow/discussions/58143>`_
+  for details.
+- **Spawn mode** (default on macOS and Windows): Workers are spawned one at a time as needed to prevent
+  the overhead of spawning many processes simultaneously.
+
+.. note::
+
+   The ``parallelism`` parameter can be configured via the ``[core] parallelism`` option in ``airflow.cfg``.
+   The default value is ``32``.
+
+.. warning::
+
+   Since LocalExecutor workers are spawned as sub-processes of the scheduler, in containerized environments
+   this may appear as excessive memory consumption by the scheduler process. This can potentially trigger
+   container restarts due to OOM (Out of Memory). Consider adjusting the ``parallelism`` value based on
+   your container's resource limits.
 
 .. note::