You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Daniel Cooper (Jira)" <ji...@apache.org> on 2019/10/03 09:40:00 UTC
[jira] [Created] (AIRFLOW-5589) KubernetesPodOperator: Duplicate
pods created on worker restart
Daniel Cooper created AIRFLOW-5589:
--------------------------------------
Summary: KubernetesPodOperator: Duplicate pods created on worker restart
Key: AIRFLOW-5589
URL: https://issues.apache.org/jira/browse/AIRFLOW-5589
Project: Apache Airflow
Issue Type: Bug
Components: worker
Affects Versions: 1.10.5, 1.10.4
Reporter: Daniel Cooper
Assignee: Daniel Cooper
K8sPodOperator holds state within the execute function that monitors the running pod. If a worker restarts for any reason (pod death, pod shuffle, upgrade etc.) then this state is lost.
At this point the scheduler notices (after max heartbeat interval wait) that the task is now 'zombie' (not monitored) and reschedules the task.
The new worker has no knowledge of the existing running pod and so creates a new duplicate pod. This can lead to many duplicate pods for the same task running together in extreme cases.
I believe this is the problem Nicholas Brenwald (King) described as having when running k8s pod operator on Google Composer (at the September meetup at King).
My fix is to add enough labels to uniquely identify a running pod as being from a given task instance (dag_id, task_id, run_id). We then do a namespaced list of pods from k8s with a label selector and monitor the existing pod if it exists otherwise we create a new one as normal.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)