You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/09/02 18:04:02 UTC
[jira] [Commented] (AIRFLOW-2027) Only trigger sleep in scheduler
after all files have parsed
[ https://issues.apache.org/jira/browse/AIRFLOW-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601514#comment-16601514 ]
Apache Spark commented on AIRFLOW-2027:
---------------------------------------
User 'aoen' has created a pull request for this issue:
https://github.com/apache/incubator-airflow/pull/2986
> Only trigger sleep in scheduler after all files have parsed
> -----------------------------------------------------------
>
> Key: AIRFLOW-2027
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2027
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
> Reporter: Dan Davydov
> Assignee: Dan Davydov
> Priority: Major
> Fix For: 1.10.0
>
>
> The scheduler loop sleeps for 1 second every loop unnecessarily. Remove this sleep to slightly speed up scheduling, and instead do it once all files have been parsed. It can add up since it runs to every scheduler loop which runs # of dags to parse/scheduler parallelism times.
> Also remove the unnecessary increased file processing interval in tests which slows them down.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)