[GitHub] [airflow] ephraimbuddy edited a comment on issue #13542: Task stuck in "scheduled" or "queued" state, pool has all slots queued, nothing is executing

[GitBox <gi...@apache.org>]

ephraimbuddy edited a comment on issue #13542:
URL: https://github.com/apache/airflow/issues/13542#issuecomment-817118815


   I observed this again last night and the reason was image pull error inside the pods. 
   
   When there's image pull error, the pod is still in 'Pending' phase and airflow sees it as such therefore tasks stay queued. If you don't do anything, after a very long time, the tasks will be cleared and the dag marked as failed.
   
   The kubernetes pod watcher does not watch the containers inside the pods. It watches only the pods and report the status as pending therefore tasks are queued.
   
   After much digging, the only solution I can think of is using timeout to delete the pods because I couldn't find a way we could watch the containers inside the pods and report the container status.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org