[jira] [Updated] (AIRFLOW-3001) Accumulative tis slow allocation of new schedule

["Jason Kim (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Kim updated AIRFLOW-3001:
-------------------------------
    Summary: Accumulative tis slow allocation of new schedule  (was: accumulative tis slow allocation of new schedule)

> Accumulative tis slow allocation of new schedule
> ------------------------------------------------
>
>                 Key: AIRFLOW-3001
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3001
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 1.10.0
>            Reporter: Jason Kim
>            Priority: Major
>
> I have created very long term schedule in short interval (2~3 years as 10 min interval)
> So, dag would be bigger and bigger as scheduling goes on.
> Finally, at critical point (I don't know exactly when it is), the allocation of new task_instances get slow and then almost stop.
> I found that in this point, many slow query logs had occurred. (I was using mysql as meta repository)
> queries like this
> "SELECT * FROM task_instance WHERE dag_id = '~' and execution_date = '~'"
> I could resolve this issue by adding new index consist of dag_id and execution_date
> So, I wanted 1.10 branch to be modified to create task_instance table with the index.
> Thanks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)