You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "JMRaczynski (via GitHub)" <gi...@apache.org> on 2026/02/10 14:49:02 UTC
[D] Should running KubernetesPodOperator pods terminate on Airflow scheduler restart? [airflow]
GitHub user JMRaczynski created a discussion: Should running KubernetesPodOperator pods terminate on Airflow scheduler restart?
Question as in title. For `KubernetesExecutor`, it is explicitly stated in [docs](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/kubernetes_executor.html#but-what-about-cases-where-the-scheduler-pod-crashes) that scheduler crashes don't lead to task failures/re-runs.
I can't find similar information about `KubernetesPodOperator`.
I currently use Airflow 3.1.3 with `apache-airflow-providers-cncf-kubernetes` 10.9.0 and `LocalExecutor`. On scheduler restart I see that KPO pods are terminated and then recreated after scheduler restart. I do not set `reattach_on_restart` parameter in my KPOs, so I expect it to be set to default value `True`.
Is it an expected behavior or a bug?
If it is expected, is it possible to prevent KPO pods in the running state from termination on scheduler restart (without hacks like custom `on_kill()` implementation?
GitHub link: https://github.com/apache/airflow/discussions/61734
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should running KubernetesPodOperator pods terminate on Airflow scheduler restart? [airflow]
Posted by "JMRaczynski (via GitHub)" <gi...@apache.org>.
GitHub user JMRaczynski edited a discussion: Should running KubernetesPodOperator pods terminate on Airflow scheduler restart?
Question as in title. For `KubernetesExecutor`, it is explicitly stated in [docs](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/kubernetes_executor.html#but-what-about-cases-where-the-scheduler-pod-crashes) that scheduler crashes don't lead to task failures/re-runs.
I can't find similar information about `KubernetesPodOperator`.
I currently use Airflow 3.1.3 with `apache-airflow-providers-cncf-kubernetes` 10.9.0 and `LocalExecutor`. On scheduler restart I see that currently running KPO pods are terminated and then recreated after scheduler restart. I do not set `reattach_on_restart` parameter in my KPOs, so I expect it to be set to default value `True`.
Is it an expected behavior or a bug?
If it is expected, is it possible to prevent KPO pods in the running state from termination on scheduler restart (without hacks like custom `on_kill()` implementation?
GitHub link: https://github.com/apache/airflow/discussions/61734
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should running KubernetesPodOperator pods terminate on Airflow scheduler restart? [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Should running KubernetesPodOperator pods terminate on Airflow scheduler restart?
> If it is expected, is it possible to prevent KPO pods in the running state from termination on scheduler restart (without hacks like custom on_kill() implementation)?
Yes. It's expected and it's consequence of using Local Executor. If you want the Pods to not be terminated you need to use Celery Executor or KubernetesExecutor to run KPOs.The reasons why it is like that is that executors are running the tasks even if KPO is used. LocalExecutor is - by definition and architectur e- restarted when you restart Scheduler and for Local Executor the tasks are "subprocesses" of the executore. In case of Celery and Kubernetes executor, tasks are running "somewwhere else" - either as separate worker subprocesses or separate k8S pods, so they "survive" scheduler restarts.
So really the docs are right - it's not the operator used that determines survivability in this case - it's the executor.
GitHub link: https://github.com/apache/airflow/discussions/61734#discussioncomment-15759792
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Should running KubernetesPodOperator pods terminate on Airflow scheduler restart? [airflow]
Posted by "JMRaczynski (via GitHub)" <gi...@apache.org>.
GitHub user JMRaczynski edited a discussion: Should running KubernetesPodOperator pods terminate on Airflow scheduler restart?
Question as in title. For `KubernetesExecutor`, it is explicitly stated in [docs](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/kubernetes_executor.html#but-what-about-cases-where-the-scheduler-pod-crashes) that scheduler crashes don't lead to task failures/re-runs.
I can't find similar information about `KubernetesPodOperator`.
I currently use Airflow 3.1.3 with `apache-airflow-providers-cncf-kubernetes` 10.9.0 and `LocalExecutor`. On scheduler restart I see that currently running KPO pods are terminated and then recreated after scheduler restart. I do not set `reattach_on_restart` parameter in my KPOs, so I expect it to be set to default value `True`.
Is it an expected behavior or a bug?
If it is expected, is it possible to prevent KPO pods in the running state from termination on scheduler restart (without hacks like custom `on_kill()` implementation)?
GitHub link: https://github.com/apache/airflow/discussions/61734
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org