[jira] [Closed] (AIRFLOW-6264) Airflow not scheduling tasks after staying in Running state

["Nidhi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-6264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nidhi closed AIRFLOW-6264.
--------------------------
    Resolution: Fixed

> Airflow not scheduling tasks after staying in Running state
> -----------------------------------------------------------
>
>                 Key: AIRFLOW-6264
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6264
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: celery, DAG, operators
>    Affects Versions: 1.10.6
>            Reporter: Nidhi
>            Priority: Major
>             Fix For: 1.10.6
>
>
> I am trying to run my tasks using Airflow and Celery Executor. I have created one DAG inside that I have approximately 60000 tasks. I am trying to run this and When I trigger a DAG it starts running but not scheduling my tasks. It stays in running state for 2 days without scheduling the tasks.
> And the main bug is If i schedule 5000 tasks then it takes less than 1 minute to schedule it but if I am trying to schedule more than that limit it is scheduling my tasks.
> I have tried different solutions also to solve this the first one is:
>  * *{{PARALLELISM=1000}}*
>  * NON_POOLED_TASK_SLOT_COUNT=1000
>  * DAG_CONCURRENCY=10000
> *But, this does not work as New Version of airflow does not support non_pooled_task_slot_count.*
> *I have tried to use CELERYD_COUNT which is also not working in my case. I have tries mostly every changes which can be useful but none of them is working for me.*
> *Can someone let me know how can schedule this much amount of tasks /*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)