[jira] [Created] (AIRFLOW-5516) Webserver crashes when there are too many dags or tasks to handle

["Vivien Morlet (Jira)" <ji...@apache.org>]

Vivien Morlet created AIRFLOW-5516:
--------------------------------------

             Summary: Webserver crashes when there are too many dags or tasks to handle
                 Key: AIRFLOW-5516
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5516
             Project: Apache Airflow
          Issue Type: Bug
          Components: DAG, ui, webserver
    Affects Versions: 1.10.2, 1.10.1, 1.10.0, 1.9.0, 1.8.2, 1.8.1
         Environment: Google Cloud Composer
            Reporter: Vivien Morlet
         Attachments: ServerError502.png, badGateway.png

Hello everyone,

 

I am using Apache Airflow with the orchestration service Composer of Google Cloud.

I have an issue with the UI Webserver. I noticed that the webserver crashes when there are too many DAGs or tasks to handle; by "crashes" I mean that the workers are still handling the tasks and DAGs are triggered but the Webserver throws 502 errors (see related screenshots).

 

I have approximately 50 DAGs that Airflow parses and there are for some DAGs sometimes thousands of tasks to handle and to schedule.

 

As I said, the workers are still doing their jobs but this is totally unconvenient (understand impossible) to handle DAGs (e.g trigger dags, show graphs etc.) without the UI Webserver especially since the service Composer of Google Cloud doesn't give a total access to every tools of Airflow.

 

Theoretically Airflow can handle an infinity of tasks but visibly the webserver suffers.
What can I do with this issue?

 

Thanks for your help, I hope that we will be able to fix this issue soon.

Regards,
Bibimorlet



--
This message was sent by Atlassian Jira
(v8.3.4#803005)