You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dheerajturaga (via GitHub)" <gi...@apache.org> on 2026/03/04 22:23:01 UTC
[PR] Centralized runtime control of Edge Worker concurrency in distributed Airflow deployments [airflow]
dheerajturaga opened a new pull request, #62896:
URL: https://github.com/apache/airflow/pull/62896
In distributed Airflow deployments using EdgeExecutor, edge workers run
at remote sites that are often unreachable from the central control
plane — behind firewalls, in air-gapped networks, or on edge computing
nodes. Prior to this change, the worker's concurrency (the number of
tasks it runs in parallel) could only be set at startup via the -c CLI
flag. Adjusting it required direct shell access to the remote machine
and a full worker restart, causing job execution downtime and defeating
the operational value of the edge architecture.
This contribution solves the problem entirely by introducing a
server-driven concurrency control mechanism that works within the
existing security and connectivity constraints of edge deployments.
Rather than requiring a new communication channel, it repurposes the
worker's existing heartbeat protocol — the only bidirectional link
between the central Airflow API server and the remote worker — to
deliver concurrency updates. An administrator runs:
`airflow edge set-worker-concurrency --edge-hostname <worker> --concurrency <n>`
This writes the desired concurrency to the central database. On the
worker's next heartbeat, the API server
returns the new value in its response, and the worker adopts it
immediately — no restart, no remote access, no downtime.
The design follows the established pattern for queue management
introduced in this provider, but extends it to a runtime-mutable
execution parameter for the first time. It required coordinated changes
across five architectural layers: the database schema (new migration),
the SQLAlchemy model, the FastAPI worker API response contract, the
async worker heartbeat loop, and the CLI command surface. The solution
preserves the core architectural guarantee of the edge executor — that
workers never hold a direct database connection — while making a
previously static configuration parameter dynamically controllable from
the central site.
This capability was made possible by the database schema versioning foundation
introduced in #61155
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
ClaudeCode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Centralized runtime control of Edge Worker concurrency in distributed deployments [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on PR #62896:
URL: https://github.com/apache/airflow/pull/62896#issuecomment-4007991611
> Cool improvement! Actually had the exact same demand today with our deployment.
>
> One remark, one question.
>
> And one additional: Would you also extend the web UI in a follow-up that similar like queues the concurrency can be adjusted in UI?
Yes! UI and docs will be updated in a follow up!
I had this ask from my userbase for a while now but I needed #61155 first to allow for this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Centralized runtime control of Edge Worker concurrency in distributed deployments [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga merged PR #62896:
URL: https://github.com/apache/airflow/pull/62896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Centralized runtime control of Edge Worker concurrency in distributed deployments [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #62896:
URL: https://github.com/apache/airflow/pull/62896#discussion_r2892203027
##########
providers/edge3/src/airflow/providers/edge3/worker_api/datamodels.py:
##########
@@ -200,3 +200,10 @@ class WorkerSetStateReturn(BaseModel):
str | None,
Field(description="Comments about the maintenance state of the worker."),
] = None
+ concurrency: Annotated[
Review Comment:
If we extend the response by another field - not being an expert in FastAPI/Pydantic - what happens if we have an running EdgeWorker < 3.2.0 and you upgrade central deployment. Will the "new" response on heartbeat be accepted or will it raise an error? Have you tested this a backwards-compat?
##########
providers/edge3/src/airflow/providers/edge3/models/db.py:
##########
@@ -31,6 +31,7 @@
_REVISION_HEADS_MAP: dict[str, str] = {
"3.0.0": "9d34dfc2de06",
+ "3.1.0": "b3c4d5e6f7a8",
Review Comment:
3.1.0 is already released, need to assume it gets to be 3.2.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Centralized runtime control of Edge Worker concurrency in distributed deployments [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on PR #62896:
URL: https://github.com/apache/airflow/pull/62896#issuecomment-4014146258
> Cool! Then once the CI errors are fixed I can make a second pass review and then LGTM!
@jscheffl done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Centralized runtime control of Edge Worker concurrency in distributed deployments [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #62896:
URL: https://github.com/apache/airflow/pull/62896#discussion_r2893876932
##########
providers/edge3/src/airflow/providers/edge3/worker_api/datamodels.py:
##########
@@ -200,3 +200,10 @@ class WorkerSetStateReturn(BaseModel):
str | None,
Field(description="Comments about the maintenance state of the worker."),
] = None
+ concurrency: Annotated[
Review Comment:
I tested the backward-compat. Edge worker shuts down cleanly with
```
2026-03-06T05:15:08.980328Z [info ] Quitting worker, signal being offline. [airflow.providers.edge3.cli.worker] loc=worker.py:290
2026-03-06T05:15:08.998516Z [info ] Version mismatch of Edge worker and Core. Quitting worker anyway
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Centralized runtime control of Edge Worker concurrency in distributed deployments [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #62896:
URL: https://github.com/apache/airflow/pull/62896#discussion_r2896925104
##########
providers/edge3/src/airflow/providers/edge3/worker_api/datamodels.py:
##########
@@ -200,3 +200,10 @@ class WorkerSetStateReturn(BaseModel):
str | None,
Field(description="Comments about the maintenance state of the worker."),
] = None
+ concurrency: Annotated[
Review Comment:
Cool!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Centralized runtime control of Edge Worker concurrency in distributed deployments [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #62896:
URL: https://github.com/apache/airflow/pull/62896#issuecomment-4012979872
Cool! Then once the CI errors are fixed I can make a second pass review and then LGTM!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org