[GitHub] [airflow] jedcunningham commented on issue #16888: Bug on KubernetesJobWatcher in Scheduler

[GitBox <gi...@apache.org>]

jedcunningham commented on issue #16888:
URL: https://github.com/apache/airflow/issues/16888#issuecomment-876779262


   @duc-luong-tnsl, we watch pods with this label:
   https://github.com/apache/airflow/blob/2.1.0/airflow/executors/kubernetes_executor.py#L129
   
   In your scheduler log, right before the traceback, what pod name was logged? Does it looks like a kubernetesexecutor worker (e.g. dagidtaskid.random)?
   https://github.com/apache/airflow/blob/2.1.0/airflow/executors/kubernetes_executor.py#L147
   
   The only time I've seen this previously is when incorrect labels were added via `pod_mutation_hook` causing KPO pods to be considered as workers in the worker adoption logic, which then lead to this traceback. See https://github.com/astronomer/airflow-chart/pull/231.
   
   Do you have a `pod_mutation_hook` that sets `kubernetes_executor`? Or happen to use the https://github.com/astronomer/airflow-chart chart?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org