You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ignacioparicio (via GitHub)" <gi...@apache.org> on 2026/08/02 13:16:05 UTC
[PR] Dispatch the highest priority tasks first in the executor [airflow]
ignacioparicio opened a new pull request, #70942:
URL: https://github.com/apache/airflow/pull/70942
`BaseExecutor` sorts its queued tasks by `priority_weight` ascending, then dispatches from the front of that list. So when more tasks are queued than there are open slots, the **lowest priority tasks are the ones sent to workers**, and the highest priority ones are held back.
For example, with weights 1, 5 and 10 and one slot freeing up per loop, they run in the order 1, 5, 10; the most important task runs last.
It also does not even out over time, because the queue is re-sorted on every call.
<details>
<summary> Where it happens </summary>
In `airflow-core/src/airflow/executors/base_executor.py`:
```python
# order_queued_tasks_by_priority(): lowest weight first
return sorted(self.queued_tasks.items(), key=lambda x: x[1].ti.priority_weight, reverse=False)
# _get_workloads_to_schedule(): reads from the front, stops at open_slots
for task_key, task_workload in self.order_queued_tasks_by_priority():
if len(workloads_to_schedule) >= open_slots:
break
workloads_to_schedule.append((task_key, task_workload))
```
</details>
<details>
<summary> How it got here </summary>
The sort and the way it is consumed used to match. They drifted apart in two steps:
- #61376 (`fd4fd67c0f`) switched the sort from descending to ascending and, in the same commit, changed `sorted_queue.pop(0)` to `sorted_queue.pop()`. An ascending list popped from the back still gives the highest priority first, so this was correct, and O(1) instead of O(n).
- #61153 (`9fa13c82ff`) refactored `trigger_tasks` to support callback workloads and replaced that `pop()` with the front-to-back loop above. The sort was left ascending.
</details>
This only affects an executor that has more queued tasks than open slots, so deployments that stay under their `parallelism` ceiling, or that never set `priority_weight`, see no difference either way.
What this PR does:
- `base_executor.py`: sorts descending again, and notes on the method that consumers read from the front.
- `test_base_executor.py`: adds a test with three weights and fewer open slots than tasks. The existing tests all run with more slots than tasks, so none of them reach the truncating branch.
- Adds a bugfix newsfragment.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes, Claude Code (Opus 5)
Used to trace the ordering back through the history of the two commits above, and to check that nothing else in the repo overrides the sort or depends on it being ascending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Dispatch the highest priority tasks first in the executor [airflow]
Posted by "ignacioparicio (via GitHub)" <gi...@apache.org>.
ignacioparicio commented on code in PR #70942:
URL: https://github.com/apache/airflow/pull/70942#discussion_r3786234957
##########
airflow-core/newsfragments/70942.bugfix.rst:
##########
@@ -0,0 +1 @@
+Executors now dispatch the highest ``priority_weight`` tasks first. When more tasks were queued than there were open slots, the lowest priority ones were sent to workers and the highest priority ones were held back.
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Dispatch the highest priority tasks first in the executor [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #70942:
URL: https://github.com/apache/airflow/pull/70942#discussion_r3784742632
##########
airflow-core/newsfragments/70942.bugfix.rst:
##########
@@ -0,0 +1 @@
+Executors now dispatch the highest ``priority_weight`` tasks first. When more tasks were queued than there were open slots, the lowest priority ones were sent to workers and the highest priority ones were held back.
Review Comment:
This file needs to be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Dispatch the highest priority tasks first in the executor [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #70942:
URL: https://github.com/apache/airflow/pull/70942#issuecomment-5316735412
### Backport successfully created: v3-3-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-3-test</td>
<td><a href="https://github.com/apache/airflow/pull/71715"><img src="https://img.shields.io/badge/PR-71715-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Dispatch the highest priority tasks first in the executor [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun commented on code in PR #70942:
URL: https://github.com/apache/airflow/pull/70942#discussion_r3784742632
##########
airflow-core/newsfragments/70942.bugfix.rst:
##########
@@ -0,0 +1 @@
+Executors now dispatch the highest ``priority_weight`` tasks first. When more tasks were queued than there were open slots, the lowest priority ones were sent to workers and the highest priority ones were held back.
Review Comment:
I would remove this. (we usually don't use `bugfix` newsfragment, other solution is to convert this to a significant newsfragment but that doesn't seem worth it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Dispatch the highest priority tasks first in the executor [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #70942:
URL: https://github.com/apache/airflow/pull/70942#issuecomment-5316709736
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Dispatch the highest priority tasks first in the executor [airflow]
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun merged PR #70942:
URL: https://github.com/apache/airflow/pull/70942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org