Dynamically adding DAGs to Airflow

["Saleil Bhat (BLOOMBERG/ 919 3RD A)" <sb...@bloomberg.net>]

Hey all, 

I'm new to Airflow, and I have a question concerning creating DAGs on the fly. 
I saw this snippet in the documentation: https://airflow.apache.org/docs/stable/faq.html#how-can-i-create-dags-dynamically
which suggests you can programmatically create DAGs. 

My question is, can I invoke code similar to this to create a new DAG when Airflow is already running? For example, suppose I have a DAG factory which takes some config parameters and constructs a DAG. Would it be possible to use the CLI and/or REST API to trigger a call to this DAG factory to add a new DAG to my Airflow system? 

Thanks, 
-Saleil

Re: Dynamically adding DAGs to Airflow

[Daniel Standish <dp...@gmail.com>]

Airflow will continue to periodically look for new dags when running ---
whether dynamic or otherwise.

Does your dag show up when you do airflow list_dags?  Then it will show up
in webserver sooner or later.  If it does not, then it's likely something
is wrong with your dag file.

There has been talk of changing airflow's behavior of automatically parsing
every dag over and over.  This could reduce unnecessary processing and make
"expensive" dynamic dags feasible, but I don't think this has been
implemented yet.


On Fri, May 8, 2020 at 3:55 PM Saleil Bhat (BLOOMBERG/ 919 3RD A) <
sbhat39@bloomberg.net> wrote:

> Hey all,
>
> I'm new to Airflow, and I have a question concerning creating DAGs on the
> fly.
> I saw this snippet in the documentation:
> https://airflow.apache.org/docs/stable/faq.html#how
> <#m_-1032541172678074608_m_-2727493114642542222_how>
> -can-i-create-dags-dynamically
> which suggests you can programmatically create DAGs.
>
> My question is, can I invoke code similar to this to create a new DAG when
> Airflow is already running? For example, suppose I have a DAG factory which
> takes some config parameters and constructs a DAG. Would it be possible to
> use the CLI and/or REST API to trigger a call to this DAG factory to add a
> new DAG to my Airflow system?
>
> Thanks,
> -Saleil
>
>