You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2019/05/09 05:25:53 UTC
[GitHub] [airflow] feng-tao commented on a change in pull request #5262:
[AIRFLOW-XXX] Documents about task_concurrency and pool
feng-tao commented on a change in pull request #5262: [AIRFLOW-XXX] Documents about task_concurrency and pool
URL: https://github.com/apache/airflow/pull/5262#discussion_r282342210
##########
File path: docs/faq.rst
##########
@@ -172,12 +172,13 @@ There are many layers of ``airflow run`` commands, meaning it can call itself.
How can my airflow dag run faster?
----------------------------------
-There are three variables we could control to improve airflow dag performance:
+There are a few variables we could control to improve airflow dag performance:
-- ``parallelism``: This variable controls the number of task instances that the airflow worker can run simultaneously. User could increase the parallelism variable in the ``airflow.cfg``.
-- ``concurrency``: The Airflow scheduler will run no more than ``$concurrency`` task instances for your DAG at any given time. Concurrency is defined in your Airflow DAG. If you do not set the concurrency on your DAG, the scheduler will use the default value from the ``dag_concurrency`` entry in your ``airflow.cfg``.
+- ``parallelism``: This variable controls the number of task instances that runs simultaneously across the whole Airflow cluster. User could increase the parallelism variable in the ``airflow.cfg``.
+- ``concurrency``: The Airflow scheduler will run no more than ``concurrency`` task instances for your DAG at any given time. Concurrency is defined in your Airflow DAG. If you do not set the concurrency on your DAG, the scheduler will use the default value from the ``dag_concurrency`` entry in your ``airflow.cfg``.
+- ``task_concurrency``: This variable controls the number of concurrent running task instances across ``execution_dates`` per task.
Review comment:
across all the dag runs or across execution dates?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services