[jira] [Created] (AIRFLOW-1945) Pass --autoscale to celery workers

["Michael O. (JIRA)" <ji...@apache.org>]

Michael O. created AIRFLOW-1945:
-----------------------------------

             Summary: Pass --autoscale to celery workers
                 Key: AIRFLOW-1945
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1945
             Project: Apache Airflow
          Issue Type: Improvement
          Components: celery, cli
            Reporter: Michael O.
            Priority: Trivial


Celery supports autoscaling of the worker pool size (number of tasks that can parallelize within one worker node).  I'd like to propose to support passing the --autoscale parameter to {{airflow worker}}.
Since this is a trivial change, I am not sure if there's any reason for not being supported already.(?)

For example
{{airflow worker --concurrency=4}} will set a fixed pool size of 4.

With minimal changes in [https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855] it could support
{{airflow worker --autoscale=2,10}} to set an autoscaled pool size of 2 to 10


Some references:
* http://docs.celeryproject.org/en/latest/internals/reference/celery.worker.autoscale.html
* https://github.com/apache/incubator-airflow/blob/4ce4faaeae7a76d97defcf9a9d3304ac9d78b9bd/airflow/bin/cli.py#L855



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)