[jira] [Updated] (AIRFLOW-3001) Accumulative tis slow allocation of new schedule

["Ash Berlin-Taylor (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ash Berlin-Taylor updated AIRFLOW-3001:
---------------------------------------
    Fix Version/s:     (was: 2.0.0)
                   1.10.2

> Accumulative tis slow allocation of new schedule
> ------------------------------------------------
>
>                 Key: AIRFLOW-3001
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3001
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 1.10.0
>            Reporter: Jason Kim
>            Assignee: Jason Kim
>            Priority: Major
>             Fix For: 1.10.2
>
>
> I have created very long term schedule in short interval. (2~3 years as 10 min interval)
> So, dag could be bigger and bigger as scheduling goes on.
> Finally, at critical point (I don't know exactly when it is), the allocation of new task_instances get slow and then almost stop.
> I found that in this point, many slow query logs had occurred. (I was using mysql as meta repository)
> queries like this
> "SELECT * FROM task_instance WHERE dag_id = 'some_dag_id' AND execution_date = ''2018-09-01 00:00:00"
> I could resolve this issue by adding new index consists of dag_id and execution_date.
> So, I wanted 1.10 branch to be modified to create task_instance table with the index.
> Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)