You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/10 21:57:27 UTC
[GitHub] [airflow] jedcunningham opened a new pull request, #28844: Annotate KubeExecutor pods that we don't delete
jedcunningham opened a new pull request, #28844:
URL: https://github.com/apache/airflow/pull/28844
We weren't keeping track of which pods we'd finished with yet, so if you had `[kubernetes_executor] delete_worker_pods` false, your KubeExecutor would adopt every single remaining pod when starting up. Every time.
We now annotate them with `airflow_done` when processing a pods event from the watcher, so we can ignore the pod when doing adoption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #28844: Annotate KubeExecutor pods that we don't delete
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #28844:
URL: https://github.com/apache/airflow/pull/28844#discussion_r1067402778
##########
airflow/executors/kubernetes_executor.py:
##########
@@ -52,6 +52,7 @@
from airflow.utils.state import State
ALL_NAMESPACES = "ALL_NAMESPACES"
+POD_DONE_KEY = "airflow_done"
Review Comment:
Thanks, renamed some stuff based on this. Take another look please 🙏.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #28844: Annotate KubeExecutor pods that we don't delete
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #28844:
URL: https://github.com/apache/airflow/pull/28844#discussion_r1066432769
##########
airflow/executors/kubernetes_executor.py:
##########
@@ -52,6 +52,7 @@
from airflow.utils.state import State
ALL_NAMESPACES = "ALL_NAMESPACES"
+POD_DONE_KEY = "airflow_done"
Review Comment:
Happy to adjust naming if there are better ideas.
##########
airflow/executors/kubernetes_executor.py:
##########
@@ -367,6 +368,18 @@ def delete_pod(self, pod_id: str, namespace: str) -> None:
if e.status != 404:
raise
+ def patch_done_with_pod(self, *, pod_id: str, namespace: str):
Review Comment:
Same here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham merged pull request #28844: Annotate KubeExecutor pods that we don't delete
Posted by GitBox <gi...@apache.org>.
jedcunningham merged PR #28844:
URL: https://github.com/apache/airflow/pull/28844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28844: Annotate KubeExecutor pods that we don't delete
Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #28844:
URL: https://github.com/apache/airflow/pull/28844#discussion_r1066657098
##########
airflow/executors/kubernetes_executor.py:
##########
@@ -52,6 +52,7 @@
from airflow.utils.state import State
ALL_NAMESPACES = "ALL_NAMESPACES"
+POD_DONE_KEY = "airflow_done"
Review Comment:
just to shed it up...
"executor_done" or "do_not_adopt"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org