[GitHub] [airflow] Jorricks commented on issue #13542: Task stuck in "scheduled" or "queued" state, pool has all slots queued, nothing is executing

[GitBox <gi...@apache.org>]

Jorricks commented on issue #13542:
URL: https://github.com/apache/airflow/issues/13542#issuecomment-924363913


   > We're continuing to see an issue in `2.1.3` where DAG tasks appear to be running for many hours, even days, but are making no visible progress. This situation persists through scheduler restarts and is not resolved until we clear each stuck task manually twice. The first time we clear the task we see an odd rendering in the UI with a dark blue border that looks like this:
   > 
   > <img alt="Screen Shot 2021-09-21 at 7 40 43 AM" width="637" src="https://user-images.githubusercontent.com/74351/134193377-7f826066-37cc-4f48-bb6f-b2b2224a6be7.png">
   > 
   > The second time we clear the task the border changes to light green and it usually completes as expected.
   > 
   > This is a pretty frustrating situation because the only known path to remediation is manual intervention. As previously stated we're deploying to ECS, with each Airflow process as its own ECS service in an ECS cluster; this generally works well except as noted here.
   
   Hey @maxcountryman,
   
   That sounds very annoying. Sorry to hear that.
   I guess you are using the KubernetesExecutor?
   Next time that happens could you do the following two things:
   1. Send a USR2 kill command to the scheduler and list the output here. Example: `pkill -f -USR2 "airflow scheduler"`
   2. Check the TaskInstance that is stuck and especially the `external executor id` attribute of the TaskInstance. You can find this on the detailed view of a task instance.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org