[jira] [Commented] (AIRFLOW-3607) Decreasing scheduler delay between tasks

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126628#comment-17126628 ] 

ASF GitHub Bot commented on AIRFLOW-3607:
-----------------------------------------

ashb commented on pull request #7503:
URL: https://github.com/apache/airflow/pull/7503#issuecomment-639392584


   @mik-laj Is it worth adding one of your query-count style tests here do you think?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Decreasing scheduler delay between tasks
> ----------------------------------------
>
>                 Key: AIRFLOW-3607
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3607
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 1.10.0, 1.10.1, 1.10.2
>         Environment: ubuntu 14.04
>            Reporter: Amichai Horvitz
>            Assignee: Amichai Horvitz
>            Priority: Major
>             Fix For: 2.0.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) that says instead of checking the query for every task let the tasks report to the dagrun. I have a dag with many tasks and the delay between tasks can rise to 10 seconds or more, I already changed the configuration, added processes and memory, checked the code and did research, profiling and other experiments. I hope that this change will make a drastic change in the delay. I would be happy to discuss this solution, the research and other solutions for this issue.  
> Thanks



--
This message was sent by Atlassian Jira
(v8.3.4#803005)