[GitHub] [airflow] nadflinn commented on pull request #8695: Make celery worker_prefetch_multiplier configurable

[GitBox <gi...@apache.org>]

nadflinn commented on pull request #8695:
URL: https://github.com/apache/airflow/pull/8695#issuecomment-624770541


   Yeah, I had opened this before the celery fix was accepted.  I'm ambivalent about it at this point. I wouldn't say this is necessarily "fixing" an external issue - it's just exposing `worker_prefetch_multiplier` so that it can be configured.  I added some language in the description of `worker_prefetch_multiplier` that was specific to the bug and that should probably be removed.  But it is an optimization setting that users could want to adjust independent of it being a work-around for this bug.  That said, it is an optimization ideal for lots of short running tasks where having the tasks prefetched is advantageous, which doesn't really seem to overlap with how Airflow is generally used.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org