You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jscheffl (via GitHub)" <gi...@apache.org> on 2025/10/30 18:56:52 UTC

[PR] Remove leftovers from unlimited parallelism in local executor [airflow]

jscheffl opened a new pull request, #57579:
URL: https://github.com/apache/airflow/pull/57579

   closes: #57495
   
   In Airflow 3.0.0 in PR https://github.com/apache/airflow/pull/48218 we removed unlimited scheduling. But it was reported that docs and leftovers in code of LocalExecutor still refer and implement leftovers of the legacy function.
   
   This PR cleans the documentation and removes leftovers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove leftovers from unlimited parallelism in local executor [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #57579:
URL: https://github.com/apache/airflow/pull/57579#discussion_r2479706907


##########
airflow-core/docs/core-concepts/executor/local.rst:
##########
@@ -21,32 +21,29 @@
 Local Executor
 ==============
 
-:class:`~airflow.executors.local_executor.LocalExecutor` runs tasks by spawning processes in a controlled fashion in different modes.
+:class:`~airflow.executors.local_executor.LocalExecutor` runs tasks by spawning processes in a controlled fashion on the scheduler node.
 
-Given that BaseExecutor has the option to receive a ``parallelism`` parameter to limit the number of process spawned,
-when this parameter is ``0`` the number of processes that LocalExecutor can spawn is unlimited.
+The parameter ``parallelism`` limits the number of process spawned not to overwhelm the node.
+This parameter must be greater than ``0``.

Review Comment:
   Same as below (hope this is now not making a circular dependency 8-) )



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove leftovers from unlimited parallelism in local executor [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #57579:
URL: https://github.com/apache/airflow/pull/57579#discussion_r2482283004


##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -138,7 +138,7 @@ class LocalExecutor(BaseExecutor):
 
     It uses the multiprocessing Python library and queues to parallelize the execution of tasks.
 
-    :param parallelism: how many parallel processes are run in the executor
+    :param parallelism: how many parallel processes are run in the executor, must be > 0

Review Comment:
   Ah! My mistake, looks good :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove leftovers from unlimited parallelism in local executor [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #57579:
URL: https://github.com/apache/airflow/pull/57579#discussion_r2479705570


##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -138,7 +138,7 @@ class LocalExecutor(BaseExecutor):
 
     It uses the multiprocessing Python library and queues to parallelize the execution of tasks.
 
-    :param parallelism: how many parallel processes are run in the executor
+    :param parallelism: how many parallel processes are run in the executor, must be > 0

Review Comment:
   Looking at https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/executors/base_executor.py#L177-L178 the value 0 is rejected since PR https://github.com/apache/airflow/pull/48218 and also according to devlist thread https://lists.apache.org/thread/jlroywp6tlcrgdsx28shl1nkll8bpvbt



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove leftovers from unlimited parallelism in local executor [airflow]

Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #57579:
URL: https://github.com/apache/airflow/pull/57579#discussion_r2479297652


##########
airflow-core/docs/core-concepts/executor/local.rst:
##########
@@ -21,32 +21,29 @@
 Local Executor
 ==============
 
-:class:`~airflow.executors.local_executor.LocalExecutor` runs tasks by spawning processes in a controlled fashion in different modes.
+:class:`~airflow.executors.local_executor.LocalExecutor` runs tasks by spawning processes in a controlled fashion on the scheduler node.
 
-Given that BaseExecutor has the option to receive a ``parallelism`` parameter to limit the number of process spawned,
-when this parameter is ``0`` the number of processes that LocalExecutor can spawn is unlimited.
+The parameter ``parallelism`` limits the number of process spawned not to overwhelm the node.
+This parameter must be greater than ``0``.

Review Comment:
   Base executor init will still allow parallelism to be equal to zero. Which I suppose you may want, to disable an executor from taking tasks? Not sure I really think that's a valid use case but the code does allow it, so we should either change that or update this:
   
   
   ```suggestion
   This parameter must be greater than or equal to ``0``.
   ```



##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -138,7 +138,7 @@ class LocalExecutor(BaseExecutor):
 
     It uses the multiprocessing Python library and queues to parallelize the execution of tasks.
 
-    :param parallelism: how many parallel processes are run in the executor
+    :param parallelism: how many parallel processes are run in the executor, must be > 0

Review Comment:
   Same as above:
   
   
   ```suggestion
       :param parallelism: how many parallel processes are run in the executor, must be >= 0
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove leftovers from unlimited parallelism in local executor [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #57579:
URL: https://github.com/apache/airflow/pull/57579


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove leftovers from unlimited parallelism in local executor [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #57579:
URL: https://github.com/apache/airflow/pull/57579#issuecomment-3474276888

   ### Backport failed to create: v3-1-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/18981384915'> Run details </a>
   
   <table>
       <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>❌</td>
           <td>v3-1-test</td>
           <td><a href="https://github.com/apache/airflow/commit/663d2c686580dfe402d4c2fe46781dccab1b5889"><img src='https://img.shields.io/badge/Commit-663d2c6-red' alt='Commit Link'></a></td>
       </tr>
   </table>
   
   You can attempt to backport this manually by running:
   
   ```bash
   cherry_picker 663d2c6 v3-1-test
   ```
   
   This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
   the files that need manual conflict resolution.
   
   After you have resolved the conflicts, you can continue the backport process by running:
   
   ```bash
   cherry_picker --continue
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org