You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/08/19 16:19:40 UTC
[GitHub] [airflow] potiuk commented on pull request #31829: Disable celery `task_acks_late`
potiuk commented on PR #31829:
URL: https://github.com/apache/airflow/pull/31829#issuecomment-1685038036
> > The basic meaning of this config is:
> > task_acks_late=True: celery workers [acknowledge tasks LATE](https://docs.celeryq.dev/en/stable/glossary.html#term-late-acknowledgment), AFTER they finish running
> > task_acks_late=False: celery workers [acknowledge tasks EARLY](https://docs.celeryq.dev/en/stable/glossary.html#term-early-acknowledgment), just BEFORE they start running
>
> acknowledged in celery = marking it as succeeded Currently when the celery worker goes down, we get an exception "task killed externally". But after implementing the proposed change, it is unclear whether the scheduler will mark the task as succeeded or retry it. Further investigation or testing is required to determine the exact behavior in this scenario.
I think a good way of approaching it and seeing if it makes sense would be to **finally** describe in Celery Executor the sequence diagrams showing the various cases and to make sure that all the different cases are well covered and see how "late/early ack" influences it. We badly miss it now to be able to reason about .
Not sure who can do it, but maybe somoene who wants to learn more about Airflow could do such in-depth analysis and write such diagrams, reverse engineering what our-code + celery does it? Or maybe someone who alreayd got such an in-depth analysis could do it?
I think without such visualisaation and thinking through all the use cases, changing what works mostly for many people is quite risky (I am very pro improving and changing things but only when it is accompanied by understanding what we are doing).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org