You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Vineet Goel <vi...@robinhood.com> on 2016/06/28 07:53:38 UTC
CeleryExecutor and SchedulerJob: task db state update on completion
Hi Team Airflow,
While exploring the codebase to get a deeper understanding of how airflow
works, I could find where the celery executor (worker process) marks a task
done in its local context (in the event_buffer). For example, with success
here:
https://github.com/apache/incubator-airflow/blob/master/airflow/executors/celery_executor.py#L91
I further saw that this results in the BaseExecutor marking the task done
in its event_buffer here:
https://github.com/apache/incubator-airflow/blob/master/airflow/executors/base_executor.py#L38
I could find where the BackfillJob commits the state of the tasks to the db
but I couldn't find where the SchedulerJob does this. Could anyone point me
to the part in the codebase where the SchedulerJob commits the states of
its jobs to the db.
Thanks,
Vineet