[GitHub] [airflow] val2k commented on issue #10790: Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

[GitBox <gi...@apache.org>]

val2k commented on issue #10790:
URL: https://github.com/apache/airflow/issues/10790#issuecomment-1012934214


   We face the same issue with tasks that stay indefinitely in a queued status, except that we don't see tasks as `up_for_retry`. It happens randomly within our DAGs. The task will stay in a queued status forever until we manually make it fail. We **don't use any sensors** at all. We are on an AWS MWAA instance (Airflow 2.0.2).
   
   Example logs:
   Scheduler:
   ```
   [2022-01-14 08:03:32,868] {{scheduler_job.py:1239}} ERROR - Executor reports task instance <TaskInstance: task0 2022-01-13 07:00:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
   [2022-01-14 08:03:32,845] {{scheduler_job.py:1210}} INFO - Executor reports execution of task0 execution_date=2022-01-13 07:00:00+00:00 exited with status failed for try_number 1
   <TaskInstance: task0 2022-01-13 07:00:00+00:00 [queued]> in state FAILURE
   ```
   
   Worker:
   `[2021-04-20 20:54:29,109: ERROR/ForkPoolWorker-15] Failed to execute task dag_id could not be found: task0. Either the dag did not exist or it failed to parse..`
   This is not seen in the worker logs for every occurrence in the scheduler logs.
   
   Because of the MWAA autoscaling mechanism, `worker_concurrency` is not configurable.
   `worker_autoscale`: `10, 10`.
   `dagbag_import_timeout`: 120s
   `dag_file_processor_timeout`: 50s
   `parallelism` = 48
   `dag_concurrency` = 10000
   `max_threads` = 8
   
   We currently have 2 (minWorkers) to 10 (maxWorkers) mw1.medium (2 vCPU) workers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org