[GitHub] [airflow] potiuk edited a comment on issue #5594: [AIRFLOW-4924] Loading DAGs asynchronously in Airflow webserver

[GitBox <gi...@apache.org>]

potiuk edited a comment on issue #5594: [AIRFLOW-4924] Loading DAGs asynchronously in Airflow webserver
URL: https://github.com/apache/airflow/pull/5594#issuecomment-511661597
 
 
   And I would not worry about multiprocessing queue problems in this case. In this case we have simple Producer -> Consumer case and we use it in the simplest possible way -> put() in producer and get() in consumer. The problems with multiprocessing queues that we experienced were related to rather complex pattern of using the queues where we had several queues synchronising back-forth and we were using length of the queue as indication of whether there is anything left for processing. 
   
   ~I would however possibly think about increasing the maxsize of the Queue (will leave a comment there) to increase parallelism.~
   Scrap that. I thought a bit and maxsize=1 is a good idea :).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services