[GitHub] [airflow] jpradass commented on issue #19767: dag callbacks are missed sometimes

[GitBox <gi...@apache.org>]

jpradass commented on issue #19767:
URL: https://github.com/apache/airflow/issues/19767#issuecomment-989749279


   > I had a look at the airflow sheduler logs and found this:
   > 
   > ```
   > ./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:15:52,278] {manager.py:1092} ERROR - Processor for some_dag.py with PID 250 started at 2021-12-08T14:15:01.525929+00:00 has timed out, killing it.
   > ./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:16:43,632] {manager.py:1092} ERROR - Processor for some_dag.py with PID 581 started at 2021-12-08T14:15:53.491981+00:00 has timed out, killing it.
   > ./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:17:35,850] {manager.py:1092} ERROR - Processor for some_dag.py with PID 898 started at 2021-12-08T14:16:45.152570+00:00 has timed out, killing it.
   > ./logs/dag_processor_manager/dag_processor_manager.log:[2021-12-08 22:18:28,167] {manager.py:1092} ERROR - Processor for some_dag.py with PID 1114 started at 2021-12-08T14:17:37.449065+00:00 has timed out, killing it.
   > ```
   > 
   > While dag callbacks are executed by the processor as the source code indicated(in the process_file function), this timeout error could lead to callbacks missed.
   > 
   > I set up this [config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-file-processor-timeout) (to 100 seconds or more) and this problem seems to be fixed.
   > 
   > I think it's a not a good idea to execute callback while file procesing. Maybe, use a specialized process or the executor to handle callback is better. (for airflowers :) )
   > 
   > @jpradass
   
   I'll give it a try and check if that works for me too. 
   
   Thank you @ghostbody 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org