[GitHub] yrqls21 edited a comment on issue #3830: [AIRFLOW-2156] Parallelize Celery Executor task state fetching

[GitBox <gi...@apache.org>]

yrqls21 edited a comment on issue #3830: [AIRFLOW-2156] Parallelize Celery Executor task state fetching
URL: https://github.com/apache/incubator-airflow/pull/3830#issuecomment-419695750
 
 
   Thanks @ashb for pointing out,  I've changed the commit title and PR title to `Parallelize Celery Executor task state fetching` to better reflect the content of the change. Also open for discussion on the default value of `sync_parallelism`( given the small difference different value can cause let's keep it short). I vote for `16`, it has proven to provide great performance, ~1s running time for querying 30k running task states( of course this also heavily depends on the network, celery broker, backend etc, we use redis for broker and backend), and is not a crazy number that will create much burden on the box.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services