[GitHub] [airflow] yuqian90 edited a comment on issue #16163: Confusing log for long running tasks: "dependency 'Task Instance Not Running' FAILED: Task is in the running state"

[GitBox <gi...@apache.org>]

yuqian90 edited a comment on issue #16163:
URL: https://github.com/apache/airflow/issues/16163#issuecomment-868977040


   After some more investigation, it's very likely we see this log appearing an hour after a long running task started because of the default `visibility_timeout` setting in Celery. This code in default_celery.py sets `visibility_timeout` to 21600 only if the broker_url starts with redis or sql. In our case we are using redis sentinels so it's still redis although the URL starts with `sentinel`. Therefore the `visibility_timeout` is left at 3600 which is the default according to [celery documentation](https://docs.celeryproject.org/en/v4.4.7/getting-started/brokers/redis.html#visibility-timeout). The weird thing is that after I tried to manually change `visibility_timeout` to a very large integer in airflow.cfg, the same log still showed up exactly an hour after a task started. So it seems changing `visibility_timeout` in this case does not make any difference. Not sure if anyone experienced the same.
   
   @david30907d maybe try changing `visibility_timeout` to a large number in your setup and see if it still happens after an hour. If it stops for you, it means `visibility_timeout` is probably the cause. There may be something wrong in our own setup causing changing `visibility_timeout` not to take effect.
   
   ```python
   def _broker_supports_visibility_timeout(url):
       return url.startswith("redis://") or url.startswith("sqs://")
   
   
   log = logging.getLogger(__name__)
   
   broker_url = conf.get('celery', 'BROKER_URL')
   
   broker_transport_options = conf.getsection('celery_broker_transport_options') or {}
   if 'visibility_timeout' not in broker_transport_options:
       if _broker_supports_visibility_timeout(broker_url):
           broker_transport_options['visibility_timeout'] = 21600
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org