You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jack (JIRA)" <ji...@apache.org> on 2019/04/23 15:05:00 UTC
[jira] [Updated] (AIRFLOW-3145) reduce airflow dag scheduling
latency in production
[ https://issues.apache.org/jira/browse/AIRFLOW-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
jack updated AIRFLOW-3145:
--------------------------
Description:
The docs says:
* {{}}
{code:java}
max_threads: Scheduler will spawn multiple threads in parallel to schedule dags. This is controlled by max_threads with default value of 2. User should increase this value to a larger value(e.g numbers of cpus where scheduler runs - 1) in production.{code}
[https://airflow.apache.org/faq.html#how-can-my-airflow-dag-run-faster]
However in the Aiflow.cfg the comment is
{code:java}
#The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run.
max_threads = 2{code}
May I suggest that the
{code:java}
User should increase this value to a larger value(e.g numbers of cpus where scheduler runs - 1) in production.{code}
will also be written in the comment at Aiflow.cfg
Not everyone read the docs from top to bottom this could be easily missed.
was:
The docs says:
* {{}}
{code:java}
max_threads: Scheduler will spawn multiple threads in parallel to schedule dags. This is controlled by max_threads with default value of 2. User should increase this value to a larger value(e.g numbers of cpus where scheduler runs - 1) in production.{code}
[https://airflow.apache.org/faq.html#how-can-my-airflow-dag-run-faster]
However in the Aiflow.cfg the comment is
{code:java}
#The scheduler can run multiple threads in parallel to schedule dags.
# This defines how many threads will run.
max_threads = 2{code}
May I suggest that the
{code:java}
User should increase this value to a larger value(e.g numbers of cpus where scheduler runs - 1) in production.{code}
will also be written in the comment at Aiflow.cfg
Not everyone read the docs from top to bottom this could be easily missed.
Issue Type: Improvement (was: Task)
Summary: reduce airflow dag scheduling latency in production (was: [Edit Docummntation ] reduce airflow dag scheduling latency in production)
> reduce airflow dag scheduling latency in production
> ---------------------------------------------------
>
> Key: AIRFLOW-3145
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3145
> Project: Apache Airflow
> Issue Type: Improvement
> Components: docs
> Affects Versions: 1.10.1
> Reporter: jack
> Priority: Trivial
>
> The docs says:
> * {{}}
> {code:java}
> max_threads: Scheduler will spawn multiple threads in parallel to schedule dags. This is controlled by max_threads with default value of 2. User should increase this value to a larger value(e.g numbers of cpus where scheduler runs - 1) in production.{code}
> [https://airflow.apache.org/faq.html#how-can-my-airflow-dag-run-faster]
>
> However in the Aiflow.cfg the comment is
> {code:java}
> #The scheduler can run multiple threads in parallel to schedule dags.
> # This defines how many threads will run.
> max_threads = 2{code}
>
> May I suggest that the
> {code:java}
> User should increase this value to a larger value(e.g numbers of cpus where scheduler runs - 1) in production.{code}
>
> will also be written in the comment at Aiflow.cfg
> Not everyone read the docs from top to bottom this could be easily missed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)