[GitHub] [airflow] chachampis commented on issue #16163: Confusing log for long running tasks: "dependency 'Task Instance Not Running' FAILED: Task is in the running state"

[GitBox <gi...@apache.org>]

chachampis commented on issue #16163:
URL: https://github.com/apache/airflow/issues/16163#issuecomment-1145926897

   I have this problem in an AWS MWAA environment (Airflow version 2.2.2). Only it is not harmless, my task is actually getting killed after the visibility_timeout of 12 hours expires. I am not able to split the long-running task into shorter tasks (it involves a CREATE INDEX database statement that takes more than 12 hours).
   
   What I don't understand is, isn't the visibility_timeout setting [related](https://docs.celeryq.dev/en/v4.4.7/glossary.html#term-acknowledged) to celery's [task_acks_late](https://docs.celeryq.dev/en/v4.4.7/userguide/configuration.html#std:setting-task_acks_late) setting? To my understanding, this is False by default so a task is acknowledged just before its execution starts. If that is true, then why is the message picked up by another worker after the visibility_timeout has expired?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org