[GitHub] [airflow] hterik commented on issue #24466: Long running tasks being killed with CeleryKubernetesExecutor

[GitBox <gi...@apache.org>]

hterik commented on issue #24466:
URL: https://github.com/apache/airflow/issues/24466#issuecomment-1271195721

   Also seeing similar issues with CeleryKuberenetesExecutor on 2.3.1. Tasks on the celery worker randomly just abort halfway through with following message in the log
   ```
   2022-07-15 13:25:27 I/task     Dependencies not met for <TaskInstance: xxxxxxxxx [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state. 
   2022-07-15 13:25:27 I/task     Dependencies not met for <TaskInstance: xxxxxxxxx [running]>, dependency 'Task Instance Not Running' FAILED: Task is in the running state 
   ```
   
   Only running one scheduler.
   Unfortunately also missing the scheduler logs around those times. 
   One more unusual thing we do is occasionally send `celery.control.broadcast("shutdown")` to auto-update the worker container, this _should_ wait for all tasks to complete and again sorry to be missing logs to correlate if those can be related.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org