[GitHub] [airflow] ghostbody commented on issue #10790: Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

[GitBox <gi...@apache.org>]

ghostbody commented on issue #10790:
URL: https://github.com/apache/airflow/issues/10790#issuecomment-1058801920


   After STRUGLING, We found a method to 100% reproduce this issue !!!
   
   
   tl;dr 
   
   https://github.com/apache/airflow/blob/9ac742885ffb83c15f7e3dc910b0cf9df073407a/airflow/models/taskinstance.py#L1253
   
   Add a `raise` to simulate  db error which will likely happen when the DB is under great pressure.
   
   Then you will get this issue `Was the task killed externally` in all the time.
   
   Conditions:
   
   - Airflow 2.2
   - Celery Executor
   
   It's becasue the worker use a local task job which will spwan a child process to execute the job. The parent process set the task from `Queued` to `Running` State. However, when the prepare work for the parent process failed, it will lead to this error directly.
   
   related code is here: https://github.com/apache/airflow/blob/2.2.2/airflow/jobs/local_task_job.py#L89
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org