Re: [I] "Celery command failed", but "Marking task as SUCCESS" [airflow]

["EinavDanielDX (via GitHub)" <gi...@apache.org>]

EinavDanielDX commented on issue #35610:
URL: https://github.com/apache/airflow/issues/35610#issuecomment-1879997134

   I have update on the matter.
   More background regarding the deployment:
   We are using airflow with Celery Executor on Kubernetes, in addition, as we don't need the workers up and running all the time we are using Keda for scaling up and down the workers, which means workers are being killed regularly.
   Our deployment is configured to allow graceful shutdown for 4 hours ( which is more than enough for any task we run to finish its processing).
   We noticed that beside the misalignment between successful jobs and failed celery tasks we have other errors on random tasks, such as 
   "Executor reports task instance finished (success) although the task says it's queued. (Info: None) Was the task killed externally?"
   or
   "Loading metadata failed: {'success': False, 'reason': AirflowException('Task received SIGTERM signal')}"
   
   After further investigation set `AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION: "False"` 
   
   Now our flower looks better
   ![image](https://github.com/apache/airflow/assets/97506919/7de68d7e-f754-45a1-a7ab-1ad833d4e463)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org