[GitHub] [airflow] johnny0120 commented on issue #16163: Confusing log for long running tasks: "dependency 'Task Instance Not Running' FAILED: Task is in the running state"

[GitBox <gi...@apache.org>]

johnny0120 commented on issue #16163:
URL: https://github.com/apache/airflow/issues/16163#issuecomment-1231041796

   Having the same problem with Airflow 2.2.5 + celery + redis.
   
   The default visibility_timeout is actually 21600, 6 hours (not None from the airflow.cfg) , which means if the worker cannot pick up the task successfully, the task_instance will be stuck in the queued state for 6 hours and not visible to other workers. I remember when clearing tasks in this situation won't help either, may need to mark task failed first and then clear it.
   
   If I change visibility_timeout to a smaller number, say 300, 5 minutes, then due the default settings of in `airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG`, `'task_acks_late': True`, the task_instance will be killed by other workers if it keeps running after 5 minutes, complaining about the task is still in the running state. (Or I'm not sure if that is the same case when ack is lost from the worker)
   
   Can we just change the default setting of `task_acks_late` to False? Will it cause any side effects?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org