You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Charlie Griefer <cg...@platformq.com> on 2021/12/02 18:25:08 UTC
DAG default_args
Hey Folks.
We're working on our first Airflow project at my company. There were a good
number of DAG arguments that I expected we'd apply to all DAGs, so I put
them in a "default_args" variable.
It seems, however, that not all DAG args work if passed as default_args.
For example, "max_active_runs".
Some do work. It looks like the code under airflow.models.dag.py explicitly
copy "start_date" and "end_date" variables out of default_args and pass to
the DAG.
I'm curious if there's a list of valid DAG args (currently using
https://github.com/apache/airflow/blob/main/airflow/models/dag.py#L324 as a
reference), and a list of valid "default_args" args, and maybe even which
ones can go into either.
Airflow 2.1.4, if that's relevant.
Thanks!
Charlie
--
_______________________________
Charlie Griefer
Software Engineer
PlatformQ Education
www.platformqedu.com
Re: DAG default_args
Posted by Andrew Harmon <an...@gmail.com>.
I believe default_args are only for TASK level arguments. Not DAG level. So any settings you can set on a task can go in default_args.
Andrew
Sent from my iPhone.
> On Dec 2, 2021, at 1:25 PM, Charlie Griefer <cg...@platformq.com> wrote:
>
>
> Hey Folks.
>
> We're working on our first Airflow project at my company. There were a good number of DAG arguments that I expected we'd apply to all DAGs, so I put them in a "default_args" variable.
>
> It seems, however, that not all DAG args work if passed as default_args. For example, "max_active_runs".
>
> Some do work. It looks like the code under airflow.models.dag.py explicitly copy "start_date" and "end_date" variables out of default_args and pass to the DAG.
>
> I'm curious if there's a list of valid DAG args (currently using https://github.com/apache/airflow/blob/main/airflow/models/dag.py#L324 as a reference), and a list of valid "default_args" args, and maybe even which ones can go into either.
>
> Airflow 2.1.4, if that's relevant.
>
> Thanks!
> Charlie
>
> --
> _______________________________
> Charlie Griefer
> Software Engineer
>
> PlatformQ Education
> www.platformqedu.com
>
Re: DAG default_args
Posted by Charlie Griefer <cg...@platformq.com>.
Got it. Thanks, Jed and Andrew!
I'll probably implement it just like that.
Thanks again!
Charlie
On Thu, Dec 2, 2021 at 1:01 PM Jed Cunningham <je...@astronomer.io> wrote:
> Hey Charlie,
>
> As Andrew mentioned, `default_args` is intended for task level args. You'd
> be better off just unpacking your dict when you create the DAG. For example:
>
> ```python
> defaults_for_dags = {
> "max_active_runs": 1
> }
>
> with DAG(dag_id="something", **defaults_for_dags) as dag:
> ....
> ```
>
> Jed
>
> On Thu, Dec 2, 2021 at 11:25 AM Charlie Griefer <cg...@platformq.com>
> wrote:
>
>> Hey Folks.
>>
>> We're working on our first Airflow project at my company. There were a
>> good number of DAG arguments that I expected we'd apply to all DAGs, so I
>> put them in a "default_args" variable.
>>
>> It seems, however, that not all DAG args work if passed as default_args.
>> For example, "max_active_runs".
>>
>> Some do work. It looks like the code under airflow.models.dag.py
>> explicitly copy "start_date" and "end_date" variables out of default_args
>> and pass to the DAG.
>>
>> I'm curious if there's a list of valid DAG args (currently using
>> https://github.com/apache/airflow/blob/main/airflow/models/dag.py#L324
>> as a reference), and a list of valid "default_args" args, and maybe even
>> which ones can go into either.
>>
>> Airflow 2.1.4, if that's relevant.
>>
>> Thanks!
>> Charlie
>>
>> --
>>
>> _______________________________
>>
>> Charlie Griefer
>>
>> Software Engineer
>>
>> PlatformQ Education
>>
>> www.platformqedu.com
>>
>>
Re: DAG default_args
Posted by Daniel Standish <dp...@gmail.com>.
Another thing to consider is look into airflow task policy and dag policy.
These are mutation hooks that can be applied to all dags / tasks on your
cluster
On Thu, Dec 2, 2021, 12:01 PM Jed Cunningham <je...@astronomer.io> wrote:
> Hey Charlie,
>
> As Andrew mentioned, `default_args` is intended for task level args. You'd
> be better off just unpacking your dict when you create the DAG. For example:
>
> ```python
> defaults_for_dags = {
> "max_active_runs": 1
> }
>
> with DAG(dag_id="something", **defaults_for_dags) as dag:
> ....
> ```
>
> Jed
>
> On Thu, Dec 2, 2021 at 11:25 AM Charlie Griefer <cg...@platformq.com>
> wrote:
>
>> Hey Folks.
>>
>> We're working on our first Airflow project at my company. There were a
>> good number of DAG arguments that I expected we'd apply to all DAGs, so I
>> put them in a "default_args" variable.
>>
>> It seems, however, that not all DAG args work if passed as default_args.
>> For example, "max_active_runs".
>>
>> Some do work. It looks like the code under airflow.models.dag.py
>> explicitly copy "start_date" and "end_date" variables out of default_args
>> and pass to the DAG.
>>
>> I'm curious if there's a list of valid DAG args (currently using
>> https://github.com/apache/airflow/blob/main/airflow/models/dag.py#L324
>> as a reference), and a list of valid "default_args" args, and maybe even
>> which ones can go into either.
>>
>> Airflow 2.1.4, if that's relevant.
>>
>> Thanks!
>> Charlie
>>
>> --
>>
>> _______________________________
>>
>> Charlie Griefer
>>
>> Software Engineer
>>
>> PlatformQ Education
>>
>> www.platformqedu.com
>>
>>
Re: DAG default_args
Posted by Jed Cunningham <je...@astronomer.io>.
Hey Charlie,
As Andrew mentioned, `default_args` is intended for task level args. You'd
be better off just unpacking your dict when you create the DAG. For example:
```python
defaults_for_dags = {
"max_active_runs": 1
}
with DAG(dag_id="something", **defaults_for_dags) as dag:
....
```
Jed
On Thu, Dec 2, 2021 at 11:25 AM Charlie Griefer <cg...@platformq.com>
wrote:
> Hey Folks.
>
> We're working on our first Airflow project at my company. There were a
> good number of DAG arguments that I expected we'd apply to all DAGs, so I
> put them in a "default_args" variable.
>
> It seems, however, that not all DAG args work if passed as default_args.
> For example, "max_active_runs".
>
> Some do work. It looks like the code under airflow.models.dag.py
> explicitly copy "start_date" and "end_date" variables out of default_args
> and pass to the DAG.
>
> I'm curious if there's a list of valid DAG args (currently using
> https://github.com/apache/airflow/blob/main/airflow/models/dag.py#L324 as
> a reference), and a list of valid "default_args" args, and maybe even which
> ones can go into either.
>
> Airflow 2.1.4, if that's relevant.
>
> Thanks!
> Charlie
>
> --
>
> _______________________________
>
> Charlie Griefer
>
> Software Engineer
>
> PlatformQ Education
>
> www.platformqedu.com
>
>