You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@airflow.apache.org by Tucker B <ba...@gmail.com> on 2020/01/23 13:43:31 UTC

Millions of concurrent DAGs

I'm exploring open source alternatives to a bespoke architecture.
Currently, the solution is modelled with a top level DAG containing 10's of
tasks where each task is responsible for managing sub-DAGs each with 10's
of tasks. However, the number of sub-DAGs can be in the millions. I'm
currently working on prototyping a solution using Airflow but curious if
anyone might have some experience with potentially running millions of DAGs
concurrently. Or might Airflow not be a good fit for my problem? Thanks for
the help.

- Tucker