[GitHub] [airflow] uranusjr commented on a change in pull request #15037: Batch send to not overload multiprocessing pipe

[GitBox <gi...@apache.org>]

uranusjr commented on a change in pull request #15037:
URL: https://github.com/apache/airflow/pull/15037#discussion_r603567179



##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -1245,6 +1252,9 @@ def _process_executor_events(self, session: Session = None) -> int:
 
                 self.processor_agent.send_callback_to_execute(request)
 
+            if i % CALLBACK_SEND_BATCH_SIZE == 0:

Review comment:
       What do you mean by “queue it there, and then go and poll again”? Making the child connection non-blocking sounds great (I wondered briefly but couldn’t find found), but the polling loop seems to have quite a few side effects before that line, so we can’t just put the thing back and try again later. Does it mean to add a queue (list) on `DagFileProcessorManager` to store `DagParsingStat` instances when they fail to send, and add a block in the beginning of the poll loop to check and resend them first?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org