[GitHub] [airflow] ashb opened a new pull request #5615: [AIRFLOW-XXX] Use events/messages not multiprocessing.Manager

[GitBox <gi...@apache.org>]

ashb opened a new pull request #5615: [AIRFLOW-XXX] Use events/messages not multiprocessing.Manager
URL: https://github.com/apache/airflow/pull/5615
 
 
   Attempting to dig in to the issue reported by @Eronarn of leaking processes we've so far managed to track it down to a leaking multiprocess.Manager process. Somehow.
   
   So in a (vain?) attempt to sort that out I have just removed manager's from the scheduler entirely, and re-written the multiprocessing how I would if I was writing this in golang channels - passing objects/messages over a single channel, and shutting down when done (so we don't need a "Done" signal, and we can `.poll()` on the channel to see if there is anything to receive.
   
   This is a draft PR against v1-10-stable as that is what I was testing against for release purposes, but for master the diff will be similar.
   
   Does this approach seem sensible? Have I forgotten something?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services