[GitHub] [airflow] arkadiusz-bach commented on issue #21225: Tasks stuck in queued state

[GitBox <gi...@apache.org>]

arkadiusz-bach commented on issue #21225:
URL: https://github.com/apache/airflow/issues/21225#issuecomment-1035656930


   I had similar problem, it was happening, because KubernetesExecutor is picking CeleryExecutor tasks on CeleryKubernetesExecutor
   
   Celery is changing task state to queued and KubernetesExecutor to scheduled, it is happening over and over again(depends how fast task gets to running state)
   
   
   I fixed it by adding additional filter on task queue(which default to 'kubernetes' ) for KubernetesExecutor in couple of places, below is probably the one that is causing most of the problems:
   https://github.com/apache/airflow/blob/5a6a2d604979cb70c5c9d3797738f0876dd38c3b/airflow/executors/kubernetes_executor.py#L455
   
   It is taking all of the tasks in queued state, but it should take only thoose that should run on Kubernetes - has queue equaled to: 
   https://github.com/apache/airflow/blob/5a6a2d604979cb70c5c9d3797738f0876dd38c3b/airflow/config_templates/default_airflow.cfg#L743'')


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org