You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dheerajturaga (via GitHub)" <gi...@apache.org> on 2025/09/13 11:52:21 UTC
[PR] Add shutdown-all-workers command to Edge CLI [airflow]
dheerajturaga opened a new pull request, #55626:
URL: https://github.com/apache/airflow/pull/55626
Implement new Edge CLI command to gracefully shutdown all registered edge workers with confirmation prompt. Command follows similar pattern to celery shutdown-all-workers, requiring user confirmation with y/n prompt unless bypassed with --yes flag.
<img width="1737" height="335" alt="image" src="https://github.com/user-attachments/assets/06a8970b-4697-474c-8c5d-4f39b17d2a07" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add shutdown-all-workers command to Edge CLI [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #55626:
URL: https://github.com/apache/airflow/pull/55626#discussion_r2346729809
##########
providers/edge3/src/airflow/providers/edge3/cli/edge_command.py:
##########
@@ -581,4 +618,10 @@ def remove_worker_queues(args) -> None:
ARG_QUEUES_MANAGE,
),
),
+ ActionCommand(
+ name="shutdown-all-workers",
+ help=shutdown_all_workers.__doc__,
+ func=shutdown_all_workers,
+ args=(ARG_YES,),
Review Comment:
Does it make sense to add also a `--wait` option such that the command only returns when all jobs have drained and workers are down? (Do we have the same on shutdown as well or maintenance? I don't remember...)
##########
providers/edge3/src/airflow/providers/edge3/cli/edge_command.py:
##########
@@ -351,6 +351,37 @@ def remote_worker_request_shutdown(args) -> None:
logger.info("Requested shutdown of Edge Worker host %s by %s.", args.edge_hostname, getuser())
+@cli_utils.action_cli(check_db=False)
+@providers_configuration_loaded
+def shutdown_all_workers(args) -> None:
+ """Request graceful shutdown of all edge workers."""
+ _check_valid_db_connection()
+ if not (
+ args.yes
+ or input("This will shutdown all active edge workers, this cannot be undone! Proceed? (y/n)").upper()
+ == "Y"
+ ):
+ raise SystemExit("Cancelled")
+
+ from airflow.providers.edge3.models.edge_worker import get_registered_edge_hosts, request_shutdown
+
+ all_hosts = list(get_registered_edge_hosts())
+ if not all_hosts:
+ logger.info("No edge workers found to shutdown.")
+ return
+
+ shutdown_count = 0
+ for host in all_hosts:
+ try:
+ request_shutdown(host.worker_name)
Review Comment:
nit: Shutdown is also requested if the worker is already offline.
Might make sense only to request when the status is not offline and not unknown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add shutdown-all-workers command to Edge CLI [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #55626:
URL: https://github.com/apache/airflow/pull/55626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add shutdown-all-workers command to Edge CLI [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #55626:
URL: https://github.com/apache/airflow/pull/55626#discussion_r2346788719
##########
providers/edge3/src/airflow/providers/edge3/cli/edge_command.py:
##########
@@ -351,6 +351,37 @@ def remote_worker_request_shutdown(args) -> None:
logger.info("Requested shutdown of Edge Worker host %s by %s.", args.edge_hostname, getuser())
+@cli_utils.action_cli(check_db=False)
+@providers_configuration_loaded
+def shutdown_all_workers(args) -> None:
+ """Request graceful shutdown of all edge workers."""
+ _check_valid_db_connection()
+ if not (
+ args.yes
+ or input("This will shutdown all active edge workers, this cannot be undone! Proceed? (y/n)").upper()
+ == "Y"
+ ):
+ raise SystemExit("Cancelled")
+
+ from airflow.providers.edge3.models.edge_worker import get_registered_edge_hosts, request_shutdown
+
+ all_hosts = list(get_registered_edge_hosts())
+ if not all_hosts:
+ logger.info("No edge workers found to shutdown.")
+ return
+
+ shutdown_count = 0
+ for host in all_hosts:
+ try:
+ request_shutdown(host.worker_name)
Review Comment:
`request_shutdown` in providers/edge3/src/airflow/providers/edge3/models/edge_worker.py already handles this. I dont think we need to recheck in the edge_command.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add shutdown-all-workers command to Edge CLI [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #55626:
URL: https://github.com/apache/airflow/pull/55626#issuecomment-3288478055
Forgot to mention: Can you please also adjust RST docs for the maintenance commands to be consistent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add shutdown-all-workers command to Edge CLI [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #55626:
URL: https://github.com/apache/airflow/pull/55626#discussion_r2346794817
##########
providers/edge3/src/airflow/providers/edge3/cli/edge_command.py:
##########
@@ -351,6 +351,37 @@ def remote_worker_request_shutdown(args) -> None:
logger.info("Requested shutdown of Edge Worker host %s by %s.", args.edge_hostname, getuser())
+@cli_utils.action_cli(check_db=False)
+@providers_configuration_loaded
+def shutdown_all_workers(args) -> None:
+ """Request graceful shutdown of all edge workers."""
+ _check_valid_db_connection()
+ if not (
+ args.yes
+ or input("This will shutdown all active edge workers, this cannot be undone! Proceed? (y/n)").upper()
+ == "Y"
+ ):
+ raise SystemExit("Cancelled")
+
+ from airflow.providers.edge3.models.edge_worker import get_registered_edge_hosts, request_shutdown
+
+ all_hosts = list(get_registered_edge_hosts())
+ if not all_hosts:
+ logger.info("No edge workers found to shutdown.")
+ return
+
+ shutdown_count = 0
+ for host in all_hosts:
+ try:
+ request_shutdown(host.worker_name)
Review Comment:
okay, I rather was judging this being a minor nit as it tells it "requeuest" N shutdowns but actually will not if they are down anyway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add shutdown-all-workers command to Edge CLI [airflow]
Posted by "dheerajturaga (via GitHub)" <gi...@apache.org>.
dheerajturaga commented on code in PR #55626:
URL: https://github.com/apache/airflow/pull/55626#discussion_r2346785538
##########
providers/edge3/src/airflow/providers/edge3/cli/edge_command.py:
##########
@@ -581,4 +618,10 @@ def remove_worker_queues(args) -> None:
ARG_QUEUES_MANAGE,
),
),
+ ActionCommand(
+ name="shutdown-all-workers",
+ help=shutdown_all_workers.__doc__,
+ func=shutdown_all_workers,
+ args=(ARG_YES,),
Review Comment:
The behavior even for shutdown is to just request shutdown and exit. In my case it would take several hours/days to drain the jobs so I personally will never use `--wait` . We can add this later if needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org